[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 09:59:16.053855 4547 test_util.cc:268] Using random seed: -186747081
I20240426 09:59:16.064463 4547 linked_list-test.cc:100] Linked List Test Configuration:
I20240426 09:59:16.064536 4547 linked_list-test.cc:101] --------------
I20240426 09:59:16.064589 4547 linked_list-test.cc:102] 50 chains
I20240426 09:59:16.064648 4547 linked_list-test.cc:103] 3 tablets
I20240426 09:59:16.064710 4547 linked_list-test.cc:104] Mutations on
I20240426 09:59:16.064756 4547 linked_list-test.cc:105] --------------
I20240426 09:59:16.064958 4547 ts_itest-base.cc:112] Starting cluster with:
I20240426 09:59:16.065016 4547 ts_itest-base.cc:113] --------------
I20240426 09:59:16.065078 4547 ts_itest-base.cc:114] 3 tablet servers
I20240426 09:59:16.065126 4547 ts_itest-base.cc:115] 3 replicas per TS
I20240426 09:59:16.065186 4547 ts_itest-base.cc:116] --------------
2024-04-26T09:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-26T09:59:16Z Disabled control of system clock
I20240426 09:59:16.085863 4547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
/tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.112.254:40941
--webserver_interface=127.4.112.254
--webserver_port=0
--builtin_ntp_servers=127.4.112.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.112.254:40941
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240426 09:59:16.381824 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:59:16.382118 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:59:16.382257 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:59:16.389281 4559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240426 09:59:16.389389 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:59:16.389463 4559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240426 09:59:16.389540 4559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240426 09:59:16.398136 4559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.112.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.112.254:40941
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.112.254:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.4.112.254
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:59:16.399375 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:59:16.401086 4559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:59:16.410655 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:16.410686 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:16.410707 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:59:16.411403 4559 server_base.cc:1034] running on GCE node
I20240426 09:59:16.412026 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:59:16.413235 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:59:16.414430 4559 hybrid_clock.cc:648] HybridClock initialized: now 1714125556414377 us; error 34 us; skew 500 ppm
I20240426 09:59:16.414716 4559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:59:16.416815 4559 webserver.cc:460] Webserver started at http://127.4.112.254:40791/ using document root <none> and password file <none>
I20240426 09:59:16.417454 4559 fs_manager.cc:359] Metadata directory not provided
I20240426 09:59:16.417582 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:59:16.417896 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:59:16.420353 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data/instance:
uuid: "a36f40f5541f48b38d0e2585ecabaed3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.421010 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal/instance:
uuid: "a36f40f5541f48b38d0e2585ecabaed3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.425753 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240426 09:59:16.428891 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:16.430068 4559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240426 09:59:16.430212 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal
uuid: "a36f40f5541f48b38d0e2585ecabaed3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.430379 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:59:16.460219 4559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:59:16.461050 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:59:16.461313 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:59:16.481482 4559 rpc_server.cc:307] RPC server started. Bound to: 127.4.112.254:40941
I20240426 09:59:16.481567 4624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.112.254:40941 every 8 connection(s)
I20240426 09:59:16.483207 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/data/info.pb
I20240426 09:59:16.486835 4547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu as pid 4559
I20240426 09:59:16.486878 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:16.487087 4547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/master-0/wal/instance
I20240426 09:59:16.496290 4625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3: Bootstrap starting.
I20240426 09:59:16.498951 4625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:16.499863 4625 log.cc:824] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3: Log is configured to *not* fsync() on all Append() calls
I20240426 09:59:16.502099 4625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3: No bootstrap required, opened a new log
I20240426 09:59:16.510564 4625 raft_consensus.cc:348] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } }
I20240426 09:59:16.510885 4625 raft_consensus.cc:374] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:16.511005 4625 raft_consensus.cc:729] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36f40f5541f48b38d0e2585ecabaed3, State: Initialized, Role: FOLLOWER
I20240426 09:59:16.511615 4625 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } }
I20240426 09:59:16.511809 4625 raft_consensus.cc:388] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:59:16.511919 4625 raft_consensus.cc:482] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:59:16.512104 4625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:16.514389 4625 raft_consensus.cc:504] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } }
I20240426 09:59:16.514853 4625 leader_election.cc:302] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a36f40f5541f48b38d0e2585ecabaed3; no voters:
I20240426 09:59:16.515491 4625 leader_election.cc:288] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 09:59:16.515570 4630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:59:16.517051 4625 sys_catalog.cc:564] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:59:16.518154 4630 raft_consensus.cc:686] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [term 1 LEADER]: Becoming Leader. State: Replica: a36f40f5541f48b38d0e2585ecabaed3, State: Running, Role: LEADER
I20240426 09:59:16.518577 4630 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } }
I20240426 09:59:16.521701 4632 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f40f5541f48b38d0e2585ecabaed3. Latest consensus state: current_term: 1 leader_uuid: "a36f40f5541f48b38d0e2585ecabaed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } } }
I20240426 09:59:16.521996 4632 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [sys.catalog]: This master's current role is: LEADER
I20240426 09:59:16.522681 4631 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a36f40f5541f48b38d0e2585ecabaed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36f40f5541f48b38d0e2585ecabaed3" member_type: VOTER last_known_addr { host: "127.4.112.254" port: 40941 } } }
I20240426 09:59:16.522960 4631 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3 [sys.catalog]: This master's current role is: LEADER
I20240426 09:59:16.527726 4638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:59:16.532126 4638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:59:16.539400 4638 catalog_manager.cc:1347] Generated new cluster ID: 21848f7132a0437ab69517ecfc993a5c
I20240426 09:59:16.539515 4638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:59:16.563656 4638 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:59:16.564757 4638 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:59:16.574895 4638 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a36f40f5541f48b38d0e2585ecabaed3: Generated new TSK 0
I20240426 09:59:16.575659 4638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:59:16.581775 4547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
/tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.112.193:0
--local_ip_for_outbound_sockets=127.4.112.193
--webserver_interface=127.4.112.193
--webserver_port=0
--tserver_master_addrs=127.4.112.254:40941
--builtin_ntp_servers=127.4.112.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240426 09:59:16.884804 4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:59:16.885059 4649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:59:16.885175 4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:59:16.892824 4649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:59:16.892963 4649 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:59:16.893024 4649 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:59:16.893115 4649 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:59:16.893227 4649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.112.193
I20240426 09:59:16.901846 4649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.112.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.112.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.4.112.193
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.112.254:40941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.112.193
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:59:16.903034 4649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:59:16.904680 4649 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:59:16.915675 4657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:16.915696 4655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:16.915745 4654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:59:16.916435 4649 server_base.cc:1034] running on GCE node
I20240426 09:59:16.916997 4649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:59:16.917861 4649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:59:16.919062 4649 hybrid_clock.cc:648] HybridClock initialized: now 1714125556918992 us; error 64 us; skew 500 ppm
I20240426 09:59:16.919368 4649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:59:16.921383 4649 webserver.cc:460] Webserver started at http://127.4.112.193:36321/ using document root <none> and password file <none>
I20240426 09:59:16.922041 4649 fs_manager.cc:359] Metadata directory not provided
I20240426 09:59:16.922137 4649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:59:16.922426 4649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:59:16.924886 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data/instance:
uuid: "563b22b809134189a9e7b5515501a6a3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.925555 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal/instance:
uuid: "563b22b809134189a9e7b5515501a6a3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.930274 4649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240426 09:59:16.933415 4663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:16.934707 4649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240426 09:59:16.934865 4649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal
uuid: "563b22b809134189a9e7b5515501a6a3"
format_stamp: "Formatted at 2024-04-26 09:59:16 on dist-test-slave-7692"
I20240426 09:59:16.935034 4649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:59:16.952378 4649 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:59:16.953126 4649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:59:16.953387 4649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:59:16.954239 4649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:59:16.955691 4649 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:59:16.955789 4649 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:16.955883 4649 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:59:16.955953 4649 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:16.982733 4649 rpc_server.cc:307] RPC server started. Bound to: 127.4.112.193:46491
I20240426 09:59:16.982793 4775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.112.193:46491 every 8 connection(s)
I20240426 09:59:16.983958 4649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/data/info.pb
I20240426 09:59:16.985657 4547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu as pid 4649
I20240426 09:59:16.985878 4547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-0/wal/instance
I20240426 09:59:16.989235 4547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
/tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.112.194:0
--local_ip_for_outbound_sockets=127.4.112.194
--webserver_interface=127.4.112.194
--webserver_port=0
--tserver_master_addrs=127.4.112.254:40941
--builtin_ntp_servers=127.4.112.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 09:59:16.996560 4776 heartbeater.cc:347] Connected to a master server at 127.4.112.254:40941
I20240426 09:59:16.996845 4776 heartbeater.cc:464] Registering TS with master...
I20240426 09:59:16.997741 4776 heartbeater.cc:511] Master 127.4.112.254:40941 requested a full tablet report, sending...
I20240426 09:59:16.999742 4590 ts_manager.cc:194] Registered new tserver with Master: 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491)
I20240426 09:59:17.001307 4590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.112.193:55317
W20240426 09:59:17.292316 4779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:59:17.292564 4779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:59:17.292734 4779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:59:17.299892 4779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:59:17.300021 4779 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:59:17.300082 4779 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:59:17.300168 4779 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:59:17.300285 4779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.112.194
I20240426 09:59:17.308790 4779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.112.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.112.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.4.112.194
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.112.254:40941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.112.194
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:59:17.309931 4779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:59:17.311537 4779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:59:17.322108 4785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:17.322141 4787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:17.322182 4784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:59:17.323027 4779 server_base.cc:1034] running on GCE node
I20240426 09:59:17.323565 4779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:59:17.324390 4779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:59:17.325588 4779 hybrid_clock.cc:648] HybridClock initialized: now 1714125557325546 us; error 39 us; skew 500 ppm
I20240426 09:59:17.325865 4779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:59:17.331081 4779 webserver.cc:460] Webserver started at http://127.4.112.194:40425/ using document root <none> and password file <none>
I20240426 09:59:17.331737 4779 fs_manager.cc:359] Metadata directory not provided
I20240426 09:59:17.331830 4779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:59:17.332098 4779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:59:17.334517 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data/instance:
uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.335249 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal/instance:
uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.339797 4779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240426 09:59:17.342909 4793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.344163 4779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240426 09:59:17.344300 4779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal
uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.344465 4779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:59:17.362391 4779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:59:17.363098 4779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:59:17.363368 4779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:59:17.364180 4779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:59:17.365481 4779 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:59:17.365571 4779 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.365666 4779 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:59:17.365741 4779 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.393479 4779 rpc_server.cc:307] RPC server started. Bound to: 127.4.112.194:46279
I20240426 09:59:17.393543 4905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.112.194:46279 every 8 connection(s)
I20240426 09:59:17.394636 4779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/data/info.pb
I20240426 09:59:17.394927 4547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu as pid 4779
I20240426 09:59:17.395078 4547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-1/wal/instance
I20240426 09:59:17.398548 4547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
/tmp/dist-test-taskAqAFzC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.112.195:0
--local_ip_for_outbound_sockets=127.4.112.195
--webserver_interface=127.4.112.195
--webserver_port=0
--tserver_master_addrs=127.4.112.254:40941
--builtin_ntp_servers=127.4.112.212:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 09:59:17.406613 4906 heartbeater.cc:347] Connected to a master server at 127.4.112.254:40941
I20240426 09:59:17.406867 4906 heartbeater.cc:464] Registering TS with master...
I20240426 09:59:17.407611 4906 heartbeater.cc:511] Master 127.4.112.254:40941 requested a full tablet report, sending...
I20240426 09:59:17.408883 4590 ts_manager.cc:194] Registered new tserver with Master: 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279)
I20240426 09:59:17.409904 4590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.112.194:43517
W20240426 09:59:17.702698 4909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:59:17.702950 4909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:59:17.703064 4909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:59:17.710227 4909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:59:17.710350 4909 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:59:17.710393 4909 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:59:17.710450 4909 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:59:17.710525 4909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.112.195
I20240426 09:59:17.719486 4909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.112.212:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.112.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.4.112.195
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.112.254:40941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.112.195
--log_dir=/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:59:17.720714 4909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:59:17.722293 4909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:59:17.732924 4915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:17.733007 4914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:59:17.732978 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:59:17.733923 4909 server_base.cc:1034] running on GCE node
I20240426 09:59:17.734426 4909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:59:17.735292 4909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:59:17.736479 4909 hybrid_clock.cc:648] HybridClock initialized: now 1714125557736427 us; error 49 us; skew 500 ppm
I20240426 09:59:17.736752 4909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:59:17.738688 4909 webserver.cc:460] Webserver started at http://127.4.112.195:34915/ using document root <none> and password file <none>
I20240426 09:59:17.739401 4909 fs_manager.cc:359] Metadata directory not provided
I20240426 09:59:17.739492 4909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:59:17.739777 4909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:59:17.742120 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data/instance:
uuid: "bfe3ccee0ac048e6b426132b5a31254f"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.742779 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal/instance:
uuid: "bfe3ccee0ac048e6b426132b5a31254f"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.747473 4909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20240426 09:59:17.750411 4923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.751605 4909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240426 09:59:17.751749 4909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal
uuid: "bfe3ccee0ac048e6b426132b5a31254f"
format_stamp: "Formatted at 2024-04-26 09:59:17 on dist-test-slave-7692"
I20240426 09:59:17.751912 4909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:59:17.776259 4909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 09:59:17.776969 4909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:59:17.777269 4909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:59:17.778366 4909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:59:17.779716 4909 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:59:17.779805 4909 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.779902 4909 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:59:17.779968 4909 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 09:59:17.807376 4909 rpc_server.cc:307] RPC server started. Bound to: 127.4.112.195:35147
I20240426 09:59:17.807427 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.112.195:35147 every 8 connection(s)
I20240426 09:59:17.813035 4909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/data/info.pb
I20240426 09:59:17.821600 4547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAqAFzC/build/asan/bin/kudu as pid 4909
I20240426 09:59:17.821799 4547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAqAFzC/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125555987851-4547-0/linked-list-cluster/ts-2/wal/instance
I20240426 09:59:17.824976 5036 heartbeater.cc:347] Connected to a master server at 127.4.112.254:40941
I20240426 09:59:17.825218 5036 heartbeater.cc:464] Registering TS with master...
I20240426 09:59:17.826030 5036 heartbeater.cc:511] Master 127.4.112.254:40941 requested a full tablet report, sending...
I20240426 09:59:17.827275 4589 ts_manager.cc:194] Registered new tserver with Master: bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:17.828123 4589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.112.195:49781
I20240426 09:59:17.837661 4547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:59:17.857443 4589 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49306:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240426 09:59:17.863335 4589 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:59:17.890655 4969 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1df189e6e4c9289246a204323b23a (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:59:17.890650 4711 tablet_service.cc:1467] Processing CreateTablet for tablet cd33b93d8d5a4fa98623df51a9b812e9 (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:59:17.891520 4841 tablet_service.cc:1467] Processing CreateTablet for tablet cd33b93d8d5a4fa98623df51a9b812e9 (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:59:17.892076 4709 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1df189e6e4c9289246a204323b23a (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:59:17.892421 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd33b93d8d5a4fa98623df51a9b812e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.892412 4969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1df189e6e4c9289246a204323b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.892436 4711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd33b93d8d5a4fa98623df51a9b812e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.892841 4710 tablet_service.cc:1467] Processing CreateTablet for tablet eb13bec03b614847beab71dc877dccaf (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:59:17.893329 4710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb13bec03b614847beab71dc877dccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.893877 4709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1df189e6e4c9289246a204323b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.894930 4971 tablet_service.cc:1467] Processing CreateTablet for tablet cd33b93d8d5a4fa98623df51a9b812e9 (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:59:17.895484 4971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd33b93d8d5a4fa98623df51a9b812e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.896521 4970 tablet_service.cc:1467] Processing CreateTablet for tablet eb13bec03b614847beab71dc877dccaf (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:59:17.897032 4970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb13bec03b614847beab71dc877dccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.899403 4840 tablet_service.cc:1467] Processing CreateTablet for tablet eb13bec03b614847beab71dc877dccaf (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:59:17.899864 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb13bec03b614847beab71dc877dccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.890264 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1df189e6e4c9289246a204323b23a (DEFAULT_TABLE table=TestTable [id=f15f676fe1bc4b8480294462ae879fb8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:59:17.903513 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1df189e6e4c9289246a204323b23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:59:17.924117 5056 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Bootstrap starting.
I20240426 09:59:17.924468 5054 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Bootstrap starting.
I20240426 09:59:17.924997 5055 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: Bootstrap starting.
I20240426 09:59:17.927812 5055 tablet_bootstrap.cc:654] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.927860 5054 tablet_bootstrap.cc:654] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.927860 5056 tablet_bootstrap.cc:654] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.928906 5055 log.cc:824] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: Log is configured to *not* fsync() on all Append() calls
I20240426 09:59:17.928987 5056 log.cc:824] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Log is configured to *not* fsync() on all Append() calls
I20240426 09:59:17.928992 5054 log.cc:824] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Log is configured to *not* fsync() on all Append() calls
I20240426 09:59:17.931371 5055 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: No bootstrap required, opened a new log
I20240426 09:59:17.931604 5056 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: No bootstrap required, opened a new log
I20240426 09:59:17.931612 5055 ts_tablet_manager.cc:1367] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240426 09:59:17.931849 5056 ts_tablet_manager.cc:1367] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240426 09:59:17.933012 5054 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: No bootstrap required, opened a new log
I20240426 09:59:17.933250 5054 ts_tablet_manager.cc:1367] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240426 09:59:17.937743 5055 raft_consensus.cc:348] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.938062 5055 raft_consensus.cc:374] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.938179 5055 raft_consensus.cc:729] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563b22b809134189a9e7b5515501a6a3, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.938824 5055 consensus_queue.cc:260] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.940065 4776 heartbeater.cc:503] Master 127.4.112.254:40941 was elected leader, sending a full tablet report...
I20240426 09:59:17.940388 5055 ts_tablet_manager.cc:1398] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240426 09:59:17.940722 5055 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: Bootstrap starting.
I20240426 09:59:17.941488 5054 raft_consensus.cc:348] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.941921 5054 raft_consensus.cc:374] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.942013 5054 raft_consensus.cc:729] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe3ccee0ac048e6b426132b5a31254f, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.941870 5056 raft_consensus.cc:348] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.942154 5055 tablet_bootstrap.cc:654] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.942241 5056 raft_consensus.cc:374] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.942341 5056 raft_consensus.cc:729] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.942703 5054 consensus_queue.cc:260] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.942981 5056 consensus_queue.cc:260] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.949760 5036 heartbeater.cc:503] Master 127.4.112.254:40941 was elected leader, sending a full tablet report...
I20240426 09:59:17.950456 5054 ts_tablet_manager.cc:1398] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20240426 09:59:17.950498 4906 heartbeater.cc:503] Master 127.4.112.254:40941 was elected leader, sending a full tablet report...
I20240426 09:59:17.950850 5054 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f: Bootstrap starting.
I20240426 09:59:17.950966 5055 tablet_bootstrap.cc:492] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: No bootstrap required, opened a new log
I20240426 09:59:17.951097 5055 ts_tablet_manager.cc:1367] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240426 09:59:17.952046 5055 raft_consensus.cc:348] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.952311 5055 raft_consensus.cc:374] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.952451 5055 raft_consensus.cc:729] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563b22b809134189a9e7b5515501a6a3, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.952549 5054 tablet_bootstrap.cc:654] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.952790 5055 consensus_queue.cc:260] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.953348 5055 ts_tablet_manager.cc:1398] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240426 09:59:17.953366 5056 ts_tablet_manager.cc:1398] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
I20240426 09:59:17.953752 5056 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Bootstrap starting.
I20240426 09:59:17.955485 5055 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3: Bootstrap starting.
I20240426 09:59:17.956951 5055 tablet_bootstrap.cc:654] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.958046 5056 tablet_bootstrap.cc:654] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.958967 5055 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3: No bootstrap required, opened a new log
I20240426 09:59:17.959182 5055 ts_tablet_manager.cc:1367] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240426 09:59:17.959790 5056 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: No bootstrap required, opened a new log
I20240426 09:59:17.959789 5055 raft_consensus.cc:348] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.959925 5056 ts_tablet_manager.cc:1367] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240426 09:59:17.959983 5055 raft_consensus.cc:374] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.960048 5054 tablet_bootstrap.cc:492] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f: No bootstrap required, opened a new log
I20240426 09:59:17.960059 5055 raft_consensus.cc:729] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563b22b809134189a9e7b5515501a6a3, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.960177 5054 ts_tablet_manager.cc:1367] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240426 09:59:17.960286 5055 consensus_queue.cc:260] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.960813 5055 ts_tablet_manager.cc:1398] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240426 09:59:17.960855 5056 raft_consensus.cc:348] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.961035 5056 raft_consensus.cc:374] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.960951 5054 raft_consensus.cc:348] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.961108 5056 raft_consensus.cc:729] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.961117 5054 raft_consensus.cc:374] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.961206 5054 raft_consensus.cc:729] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe3ccee0ac048e6b426132b5a31254f, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.961380 5056 consensus_queue.cc:260] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.961416 5054 consensus_queue.cc:260] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.961859 5054 ts_tablet_manager.cc:1398] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240426 09:59:17.961879 5056 ts_tablet_manager.cc:1398] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240426 09:59:17.962095 5056 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Bootstrap starting.
I20240426 09:59:17.962478 5054 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f: Bootstrap starting.
I20240426 09:59:17.963560 5056 tablet_bootstrap.cc:654] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.964932 5054 tablet_bootstrap.cc:654] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f: Neither blocks nor log segments found. Creating new log.
I20240426 09:59:17.965915 5056 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: No bootstrap required, opened a new log
I20240426 09:59:17.966051 5056 ts_tablet_manager.cc:1367] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240426 09:59:17.966682 5056 raft_consensus.cc:348] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.966900 5056 raft_consensus.cc:374] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.966980 5056 raft_consensus.cc:729] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.967258 5056 consensus_queue.cc:260] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.967707 5056 ts_tablet_manager.cc:1398] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240426 09:59:17.968727 5054 tablet_bootstrap.cc:492] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f: No bootstrap required, opened a new log
I20240426 09:59:17.968864 5054 ts_tablet_manager.cc:1367] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240426 09:59:17.969506 5054 raft_consensus.cc:348] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.969722 5054 raft_consensus.cc:374] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:59:17.969801 5054 raft_consensus.cc:729] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe3ccee0ac048e6b426132b5a31254f, State: Initialized, Role: FOLLOWER
I20240426 09:59:17.970017 5054 consensus_queue.cc:260] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:17.970449 5054 ts_tablet_manager.cc:1398] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240426 09:59:17.985944 4777 tablet.cc:2369] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:59:18.052148 5060 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:59:18.052372 5060 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.053968 5060 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:18.060690 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1df189e6e4c9289246a204323b23a" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" is_pre_election: true
I20240426 09:59:18.060992 4991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1df189e6e4c9289246a204323b23a" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3ccee0ac048e6b426132b5a31254f" is_pre_election: true
I20240426 09:59:18.061234 4861 raft_consensus.cc:2454] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 0.
I20240426 09:59:18.061473 4991 raft_consensus.cc:2454] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 0.
I20240426 09:59:18.061846 4665 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
I20240426 09:59:18.062152 5060 raft_consensus.cc:2781] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:59:18.062284 5060 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:59:18.062382 5060 raft_consensus.cc:3037] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.064358 5060 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
W20240426 09:59:18.064675 5037 tablet.cc:2369] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:59:18.065032 5060 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 1 election: Requested vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:18.065351 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1df189e6e4c9289246a204323b23a" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
I20240426 09:59:18.065510 4991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1df189e6e4c9289246a204323b23a" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3ccee0ac048e6b426132b5a31254f"
I20240426 09:59:18.065536 4861 raft_consensus.cc:3037] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.065686 4991 raft_consensus.cc:3037] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.068588 4991 raft_consensus.cc:2454] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 1.
I20240426 09:59:18.068603 4861 raft_consensus.cc:2454] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 1.
I20240426 09:59:18.068945 4665 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
I20240426 09:59:18.069175 5060 raft_consensus.cc:2781] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:59:18.069552 5060 raft_consensus.cc:686] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 563b22b809134189a9e7b5515501a6a3, State: Running, Role: LEADER
I20240426 09:59:18.070036 5060 consensus_queue.cc:237] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.074265 4588 catalog_manager.cc:5484] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563b22b809134189a9e7b5515501a6a3 (127.4.112.193). New cstate: current_term: 1 leader_uuid: "563b22b809134189a9e7b5515501a6a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } health_report { overall_health: UNKNOWN } } }
I20240426 09:59:18.098284 5063 raft_consensus.cc:482] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:59:18.098455 5063 raft_consensus.cc:504] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.099972 5063 leader_election.cc:288] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:18.106091 4991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb13bec03b614847beab71dc877dccaf" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3ccee0ac048e6b426132b5a31254f" is_pre_election: true
I20240426 09:59:18.106348 4991 raft_consensus.cc:2454] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in term 0.
I20240426 09:59:18.106259 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb13bec03b614847beab71dc877dccaf" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3" is_pre_election: true
I20240426 09:59:18.106647 4731 raft_consensus.cc:2454] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in term 0.
I20240426 09:59:18.106778 4795 leader_election.cc:302] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, bfe3ccee0ac048e6b426132b5a31254f; no voters:
I20240426 09:59:18.107118 5063 raft_consensus.cc:2781] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:59:18.107262 5063 raft_consensus.cc:482] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:59:18.107358 5063 raft_consensus.cc:3037] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.109197 5063 raft_consensus.cc:504] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.109683 5063 leader_election.cc:288] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 1 election: Requested vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:18.110002 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb13bec03b614847beab71dc877dccaf" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3"
I20240426 09:59:18.110082 4991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb13bec03b614847beab71dc877dccaf" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3ccee0ac048e6b426132b5a31254f"
I20240426 09:59:18.110179 4731 raft_consensus.cc:3037] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.110280 4991 raft_consensus.cc:3037] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.112977 4991 raft_consensus.cc:2454] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in term 1.
I20240426 09:59:18.112987 4731 raft_consensus.cc:2454] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in term 1.
I20240426 09:59:18.113376 4794 leader_election.cc:302] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
I20240426 09:59:18.113653 5063 raft_consensus.cc:2781] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:59:18.114027 5063 raft_consensus.cc:686] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, State: Running, Role: LEADER
I20240426 09:59:18.114502 5063 consensus_queue.cc:237] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.118543 4588 catalog_manager.cc:5484] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194). New cstate: current_term: 1 leader_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } health_report { overall_health: UNKNOWN } } }
I20240426 09:59:18.132606 5062 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:59:18.132761 5062 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.134198 5062 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491)
I20240426 09:59:18.140192 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "bfe3ccee0ac048e6b426132b5a31254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3" is_pre_election: true
I20240426 09:59:18.140444 4731 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe3ccee0ac048e6b426132b5a31254f in term 0.
I20240426 09:59:18.140911 4924 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f; no voters:
I20240426 09:59:18.141016 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "bfe3ccee0ac048e6b426132b5a31254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" is_pre_election: true
I20240426 09:59:18.141263 4861 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe3ccee0ac048e6b426132b5a31254f in term 0.
I20240426 09:59:18.141260 5062 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:59:18.141371 5062 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:59:18.141459 5062 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.143452 5062 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.143997 5062 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [CANDIDATE]: Term 1 election: Requested vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491)
I20240426 09:59:18.144320 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "bfe3ccee0ac048e6b426132b5a31254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
I20240426 09:59:18.144436 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "bfe3ccee0ac048e6b426132b5a31254f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3"
I20240426 09:59:18.144490 4861 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.144613 4731 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:59:18.146234 4861 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe3ccee0ac048e6b426132b5a31254f in term 1.
I20240426 09:59:18.146354 4731 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe3ccee0ac048e6b426132b5a31254f in term 1.
I20240426 09:59:18.146646 4925 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, bfe3ccee0ac048e6b426132b5a31254f; no voters:
W20240426 09:59:18.146801 4907 tablet.cc:2369] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:59:18.146919 5062 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:59:18.147354 5062 raft_consensus.cc:686] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [term 1 LEADER]: Becoming Leader. State: Replica: bfe3ccee0ac048e6b426132b5a31254f, State: Running, Role: LEADER
I20240426 09:59:18.147827 5062 consensus_queue.cc:237] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:18.151847 4588 catalog_manager.cc:5484] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195). New cstate: current_term: 1 leader_uuid: "bfe3ccee0ac048e6b426132b5a31254f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } health_report { overall_health: HEALTHY } } }
I20240426 09:59:18.179316 4547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:59:18.181854 4547 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 563b22b809134189a9e7b5515501a6a3 to finish bootstrapping
I20240426 09:59:18.190261 4547 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 0e4d0ccb49a84cacb9ad7f3e2e6347a0 to finish bootstrapping
I20240426 09:59:18.196707 4547 ts_itest-base.cc:233] Waiting for 3 tablets on tserver bfe3ccee0ac048e6b426132b5a31254f to finish bootstrapping
I20240426 09:59:18.203967 5081 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.4.112.195:34915/threadz?group=all
http://127.4.112.193:36321/scans
http://127.4.112.194:40425/transactions?tablet_id=60e1df189e6e4c9289246a204323b23a
http://127.4.112.195:34915/threadz
http://127.4.112.193:36321/startup
http://127.4.112.195:34915/metrics
http://127.4.112.254:40791/tablet-servers
http://127.4.112.195:34915/tablets
http://127.4.112.254:40791/masters
http://127.4.112.193:36321/threadz?group=all
http://127.4.112.195:34915/scans
http://127.4.112.194:40425/threadz?group=all
http://127.4.112.194:40425/threadz
http://127.4.112.254:40791/threadz
http://127.4.112.193:36321/mem-trackers
http://127.4.112.194:40425/mem-trackers
http://127.4.112.254:40791/stacks
http://127.4.112.194:40425/startup
http://127.4.112.193:36321/transactions?tablet_id=60e1df189e6e4c9289246a204323b23a
http://127.4.112.194:40425/metrics
http://127.4.112.194:40425/tablets
http://127.4.112.254:40791/ipki-ca-cert
http://127.4.112.254:40791/threadz?group=all
http://127.4.112.193:36321/threadz
http://127.4.112.193:36321/metrics
http://127.4.112.254:40791/tables
http://127.4.112.254:40791/dump-entities
http://127.4.112.195:34915/maintenance-manager
http://127.4.112.193:36321/maintenance-manager
http://127.4.112.194:40425/stacks
http://127.4.112.195:34915/transactions?tablet_id=60e1df189e6e4c9289246a204323b23a
http://127.4.112.195:34915/mem-trackers
http://127.4.112.195:34915/startup
http://127.4.112.254:40791/ipki-ca-cert-pem
http://127.4.112.254:40791/ipki-ca-cert-der
http://127.4.112.254:40791/mem-trackers
http://127.4.112.194:40425/scans
http://127.4.112.254:40791/startup
http://127.4.112.193:36321/tablets
http://127.4.112.194:40425/maintenance-manager
http://127.4.112.254:40791/metrics
http://127.4.112.193:36321/stacks
http://127.4.112.195:34915/stacks
I20240426 09:59:18.208643 4547 linked_list-test-util.h:401] Running for: 30.000s
I20240426 09:59:18.208771 4547 linked_list-test-util.h:402] Sampling every 10000000 us
I20240426 09:59:18.208880 4547 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240426 09:59:18.235034 5087 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:18.242893 4991 raft_consensus.cc:1261] T eb13bec03b614847beab71dc877dccaf P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Refusing update from remote peer 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.244014 5063 consensus_queue.cc:1035] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.246450 4731 raft_consensus.cc:1261] T eb13bec03b614847beab71dc877dccaf P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Refusing update from remote peer 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.247571 5063 consensus_queue.cc:1035] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.250702 4990 raft_consensus.cc:1261] T 60e1df189e6e4c9289246a204323b23a P bfe3ccee0ac048e6b426132b5a31254f [term 1 FOLLOWER]: Refusing update from remote peer 563b22b809134189a9e7b5515501a6a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.251104 4861 raft_consensus.cc:1261] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Refusing update from remote peer 563b22b809134189a9e7b5515501a6a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.251850 5060 consensus_queue.cc:1035] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.252401 5071 consensus_queue.cc:1035] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.278134 4730 raft_consensus.cc:1261] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Refusing update from remote peer bfe3ccee0ac048e6b426132b5a31254f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.278826 4860 raft_consensus.cc:1261] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Refusing update from remote peer bfe3ccee0ac048e6b426132b5a31254f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:59:18.278913 5062 consensus_queue.cc:1035] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [LEADER]: Connected to new peer: Peer: permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.281749 5062 consensus_queue.cc:1035] T cd33b93d8d5a4fa98623df51a9b812e9 P bfe3ccee0ac048e6b426132b5a31254f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:59:18.295485 5089 mvcc.cc:205] Tried to move back new op lower bound from 7021058286689136640 to 7021058286176768000. Current Snapshot: MvccSnapshot[applied={T|T < 7021058286689136640}]
I20240426 09:59:18.297889 5091 mvcc.cc:205] Tried to move back new op lower bound from 7021058286576869376 to 7021058285858041856. Current Snapshot: MvccSnapshot[applied={T|T < 7021058286576869376 or (T in {7021058286576869376})}]
I20240426 09:59:18.303009 5090 mvcc.cc:205] Tried to move back new op lower bound from 7021058286689136640 to 7021058286176768000. Current Snapshot: MvccSnapshot[applied={T|T < 7021058286689136640 or (T in {7021058286689136640})}]
I20240426 09:59:18.437986 5082 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:19.891520 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:21.342172 5087 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:21.443770 5082 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:22.825462 5037 maintenance_manager.cc:419] P bfe3ccee0ac048e6b426132b5a31254f: Scheduling FlushMRSOp(cd33b93d8d5a4fa98623df51a9b812e9): perf score=1.000000
F20240426 09:59:22.841157 5082 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7efc7569fdbd google::LogMessage::Fail() at ??:0
@ 0x7efc756a3b83 google::LogMessage::SendToLog() at ??:0
@ 0x7efc7569f7bc google::LogMessage::Flush() at ??:0
@ 0x7efc756a0f49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efc782c7880 kudu::operator-() at ??:0
@ 0x7efc84f3939a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7efc84f0382e kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7efc84f03586 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7efc84eb0529 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7efc8797a0a7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7efc87988c5c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7efc87988ac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7efc813c96aa std::function<>::operator()() at ??:0
@ 0x7efc813c1e66 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7efc813bce71 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7efc813bafa7 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7efc813f8709 handle_request at ??:0
I20240426 09:59:22.909830 4907 maintenance_manager.cc:419] P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Scheduling FlushMRSOp(cd33b93d8d5a4fa98623df51a9b812e9): perf score=1.000000
@ 0x7efc813fadeb worker_thread at ??:0
@ 0x7efc7b7106db start_thread at ??:0
@ 0x7efc7455271f clone at ??:0
I20240426 09:59:22.996379 4777 maintenance_manager.cc:419] P 563b22b809134189a9e7b5515501a6a3: Scheduling FlushMRSOp(cd33b93d8d5a4fa98623df51a9b812e9): perf score=1.000000
W20240426 09:59:23.199586 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: recv got EOF from 127.4.112.195:35147 (error 108)
I20240426 09:59:23.205734 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:23.205883 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.213701 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.239838 4665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:23.244642 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.248435 4665 consensus_peers.cc:485] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:59:23.261198 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.284322 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.298276 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.328238 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.343530 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.375730 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.391924 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:23.395606 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.417857 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:59:23.435158 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.441169 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.448972 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.462967 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.464607 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.471338 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.496405 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.507367 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.519862 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.539501 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.542466 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.556171 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.586262 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.592962 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:23.597362 4798 maintenance_manager.cc:643] P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: FlushMRSOp(cd33b93d8d5a4fa98623df51a9b812e9) complete. Timing: real 0.687s user 0.470s sys 0.063s Metrics: {"bytes_written":143875,"cfile_init":1,"compiler_manager_pool.queue_time_us":3231,"dirs.queue_time_us":6846,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":4780,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1988,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7796,"thread_start_us":13458,"threads_started":2}
I20240426 09:59:23.599336 4907 maintenance_manager.cc:419] P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: Scheduling UndoDeltaBlockGCOp(cd33b93d8d5a4fa98623df51a9b812e9): 134581 bytes on disk
I20240426 09:59:23.600338 4798 maintenance_manager.cc:643] P 0e4d0ccb49a84cacb9ad7f3e2e6347a0: UndoDeltaBlockGCOp(cd33b93d8d5a4fa98623df51a9b812e9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240426 09:59:23.605170 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.619675 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.641939 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.662751 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.675944 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.705505 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.711434 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.732529 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.769752 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.782536 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:23.795065 4668 maintenance_manager.cc:643] P 563b22b809134189a9e7b5515501a6a3: FlushMRSOp(cd33b93d8d5a4fa98623df51a9b812e9) complete. Timing: real 0.798s user 0.459s sys 0.115s Metrics: {"bytes_written":143875,"cfile_init":1,"compiler_manager_pool.queue_time_us":18122,"dirs.queue_time_us":10819,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1666,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2104,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7796,"thread_start_us":34876,"threads_started":2}
I20240426 09:59:23.796903 4777 maintenance_manager.cc:419] P 563b22b809134189a9e7b5515501a6a3: Scheduling UndoDeltaBlockGCOp(cd33b93d8d5a4fa98623df51a9b812e9): 134581 bytes on disk
I20240426 09:59:23.797821 4668 maintenance_manager.cc:643] P 563b22b809134189a9e7b5515501a6a3: UndoDeltaBlockGCOp(cd33b93d8d5a4fa98623df51a9b812e9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240426 09:59:23.802683 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.811007 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.865360 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.896286 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.914584 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.954524 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.958279 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:23.988504 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
I20240426 09:59:24.019351 5087 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:24.035336 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.057330 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.073988 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.092317 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.158798 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.168638 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.200573 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.206882 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.247560 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:24.288425 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.291023 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:24.325860 5112 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfe3ccee0ac048e6b426132b5a31254f)
I20240426 09:59:24.325850 5131 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfe3ccee0ac048e6b426132b5a31254f)
I20240426 09:59:24.326018 5112 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:24.326037 5131 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:24.326696 5112 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:24.326803 5131 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:24.327263 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3" is_pre_election: true
I20240426 09:59:24.327518 4731 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in term 1.
I20240426 09:59:24.327903 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" is_pre_election: true
I20240426 09:59:24.328125 4861 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 1.
I20240426 09:59:24.328364 4794 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
I20240426 09:59:24.328600 5112 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:59:24.328707 5112 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfe3ccee0ac048e6b426132b5a31254f)
I20240426 09:59:24.328805 5112 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:59:24.328974 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:24.329706 4665 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
I20240426 09:59:24.330010 5131 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:59:24.330130 5131 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfe3ccee0ac048e6b426132b5a31254f)
I20240426 09:59:24.330226 5131 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 1 FOLLOWER]: Advancing to term 2
W20240426 09:59:24.330418 4665 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:24.331298 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
I20240426 09:59:24.331888 5112 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:24.332418 5112 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 election: Requested vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:24.332742 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "563b22b809134189a9e7b5515501a6a3"
W20240426 09:59:24.333663 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:24.333626 5131 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:24.334025 4731 raft_consensus.cc:2379] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in current term 2: Already voted for candidate 563b22b809134189a9e7b5515501a6a3 in this term.
I20240426 09:59:24.334306 5131 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 election: Requested vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:24.334594 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
I20240426 09:59:24.334851 4794 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:24.334926 4861 raft_consensus.cc:2379] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 563b22b809134189a9e7b5515501a6a3 in current term 2: Already voted for candidate 0e4d0ccb49a84cacb9ad7f3e2e6347a0 in this term.
I20240426 09:59:24.335350 5112 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 09:59:24.336174 4665 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:24.336293 4665 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 563b22b809134189a9e7b5515501a6a3; no voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:24.336548 5131 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 09:59:24.338157 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.418167 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.436069 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.463366 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.486562 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.551882 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.589901 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.600930 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.642836 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.700675 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.749312 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.753324 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.808420 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.853761 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.905851 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:24.924139 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
I20240426 09:59:24.949262 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:24.983830 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.019974 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.077893 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.105212 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.165849 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.195878 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.254096 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.291635 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.317211 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:25.358510 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:25.363878 5087 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:25.376339 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.438378 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.489262 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.555954 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.564625 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.597664 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:59:25.630254 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.693027 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.708027 4665 consensus_peers.cc:485] T 60e1df189e6e4c9289246a204323b23a P 563b22b809134189a9e7b5515501a6a3 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:59:25.762884 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.768538 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.905169 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.906294 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:25.977852 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.049152 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.123215 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.129240 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.204730 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:26.260087 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:26.272789 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.346514 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.359851 4690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38842: Illegal state: replica 563b22b809134189a9e7b5515501a6a3 is not leader of this config: current role FOLLOWER
I20240426 09:59:26.418977 5131 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:59:26.419255 5131 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:26.420014 5131 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:26.420538 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" is_pre_election: true
I20240426 09:59:26.420786 4861 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 2.
I20240426 09:59:26.421434 4665 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters:
W20240426 09:59:26.421736 4665 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:26.421761 5131 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 09:59:26.421885 5131 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:59:26.421995 5131 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 2 FOLLOWER]: Advancing to term 3
I20240426 09:59:26.424180 5131 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:26.424816 5131 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 election: Requested vote from peers 0e4d0ccb49a84cacb9ad7f3e2e6347a0 (127.4.112.194:46279), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
I20240426 09:59:26.425509 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd33b93d8d5a4fa98623df51a9b812e9" candidate_uuid: "563b22b809134189a9e7b5515501a6a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 238 } } ignore_live_leader: false dest_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0"
W20240426 09:59:26.425663 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:26.425722 4861 raft_consensus.cc:3037] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 2 FOLLOWER]: Advancing to term 3
W20240426 09:59:26.426172 4665 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:26.428745 4861 raft_consensus.cc:2454] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b22b809134189a9e7b5515501a6a3 in term 3.
I20240426 09:59:26.429159 4665 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0, 563b22b809134189a9e7b5515501a6a3; no voters: bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:26.429402 5131 raft_consensus.cc:2781] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 09:59:26.429867 5131 raft_consensus.cc:686] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [term 3 LEADER]: Becoming Leader. State: Replica: 563b22b809134189a9e7b5515501a6a3, State: Running, Role: LEADER
I20240426 09:59:26.430148 5131 consensus_queue.cc:237] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 238, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:26.432930 4590 catalog_manager.cc:5484] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 reported cstate change: term changed from 1 to 3, leader changed from bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195) to 563b22b809134189a9e7b5515501a6a3 (127.4.112.193). New cstate: current_term: 3 leader_uuid: "563b22b809134189a9e7b5515501a6a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } health_report { overall_health: UNKNOWN } } }
W20240426 09:59:26.438529 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:26.501758 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:26.522531 4861 raft_consensus.cc:1261] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Refusing update from remote peer 563b22b809134189a9e7b5515501a6a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 3 index: 240. (index mismatch)
W20240426 09:59:26.522863 4665 consensus_peers.cc:485] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:59:26.523221 5189 consensus_queue.cc:1035] T cd33b93d8d5a4fa98623df51a9b812e9 P 563b22b809134189a9e7b5515501a6a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240426 09:59:26.817277 5087 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240426 09:59:27.622291 5087 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f29a722bdbd google::LogMessage::Fail() at ??:0
@ 0x7f29a722fb83 google::LogMessage::SendToLog() at ??:0
@ 0x7f29a722b7bc google::LogMessage::Flush() at ??:0
@ 0x7f29a722cf49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29a9e53880 kudu::operator-() at ??:0
@ 0x7f29b6ac539a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f29b6a8f82e kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f29b6a8f586 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f29b6a3c529 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f29b95060a7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f29b9514c5c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f29b9514ac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f29b2f556aa std::function<>::operator()() at ??:0
@ 0x7f29b2f4de66 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f29b2f48e71 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f29b2f46fa7 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f29b2f84709 handle_request at ??:0
@ 0x7f29b2f86deb worker_thread at ??:0
@ 0x7f29ad29c6db start_thread at ??:0
@ 0x7f29a60de71f clone at ??:0
W20240426 09:59:27.878829 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
W20240426 09:59:27.886230 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.886230 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.900650 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.918411 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.937240 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.951603 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.978586 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:27.994130 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.025408 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.028632 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.067907 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.097033 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:59:28.105197 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.131330 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.150977 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 09:59:28.170451 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.201516 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.251530 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.283331 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.338938 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:28.357841 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:28.376160 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.439644 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.442687 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.509546 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.554755 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240426 09:59:28.576514 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.583303 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.602571 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.603725 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.622493 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.628865 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.640395 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.656134 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.695087 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.701148 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.728832 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.783746 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.785867 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.832407 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.871116 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.882495 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:28.903270 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.929819 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:28.997197 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.007107 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.038182 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.109251 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:29.110893 5247 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:29.111078 5247 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:29.111824 5247 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:29.113243 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:29.117713 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:29.117805 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:29.118060 5247 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:29.143024 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:29.163857 5247 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:29.164047 5247 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:29.164767 5247 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:29.166021 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:29.166203 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:29.166287 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:29.168658 5247 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:59:29.169979 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.251082 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.259099 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:29.306172 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:29.352119 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.368588 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.435900 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.477377 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.525592 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.591871 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.644616 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.713495 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.751897 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.836391 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.895231 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.900493 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:29.954420 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:29.966905 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.065490 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.102790 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.177767 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:30.220319 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:30.239498 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.292922 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.379001 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.421137 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:59:30.432453 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.522680 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.582710 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.662261 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.673341 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.680107 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 09:59:30.798192 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.886276 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:30.901764 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:30.938812 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:30.952616 5247 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:30.952812 5247 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:30.953588 5247 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:30.954952 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:30.955180 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:30.955272 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:30.955513 5247 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:31.046152 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.081394 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:31.183888 5247 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:31.184068 5247 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:31.184653 5247 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:31.186028 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:31.186216 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 09:59:31.186331 4794 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:31.186553 5247 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:59:31.211791 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:31.222429 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:31.227955 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.381593 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.450152 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.550709 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.603317 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.761891 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.809111 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.902168 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:31.923656 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:31.986857 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.089605 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.171020 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:32.212186 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:32.258400 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.358057 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.433542 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.549248 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.610783 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.746837 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.793064 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.907327 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 09:59:32.946015 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:32.956591 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:32.977511 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.091439 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 09:59:33.148906 5247 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:33.149118 5247 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:33.149730 5247 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:33.151278 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.151415 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:33.152105 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:33.152209 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:33.152456 5247 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:59:33.166293 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:33.206416 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:59:33.337277 5247 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:33.337505 5247 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:33.338102 5247 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:33.339305 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:33.339497 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 09:59:33.339615 4794 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:33.339821 5247 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:33.361060 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.362183 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.561811 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.573954 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.712141 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240426 09:59:33.763953 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.790326 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.968935 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:33.978745 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:34.009277 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.181737 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:34.212149 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:34.233891 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.394104 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.459723 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.614212 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.690999 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:34.837349 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.040300 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:35.042784 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.064368 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:35.221875 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:35.284426 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.344868 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 09:59:35.406903 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.532598 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.643481 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.670506 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 09:59:35.781716 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:35.783249 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.004617 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.037393 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.129987 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 09:59:36.222913 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:36.254966 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.294412 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.508100 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.557859 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.760877 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:36.827594 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:36.968711 5247 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:36.968933 5247 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:36.969771 5247 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:36.971217 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:36.971457 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:36.971589 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:36.971827 5247 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:37.019212 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.097193 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.149492 5040 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:37.167742 5247 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:37.167979 5247 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:37.168742 5247 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:37.170023 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:37.170205 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 09:59:37.170305 4794 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:37.170578 5247 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:59:37.229619 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:37.284499 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.370215 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.549592 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.649636 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.665092 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 09:59:37.825103 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:37.931561 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.096511 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.191238 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:38.209106 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240426 09:59:38.219503 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:38.225407 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:38.374475 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.512626 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.657888 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.719552 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240426 09:59:38.807416 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:38.944350 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.109007 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.222076 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:39.233070 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:39.240255 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:39.410213 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.530126 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.717192 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.980515 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:39.981330 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:40.179108 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:40.214835 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:40.273422 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:40.295132 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:40.370676 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 09:59:40.498711 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:40.604487 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240426 09:59:40.613329 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:40.820220 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:40.937824 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:41.146104 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:41.213264 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:41.265525 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:41.309964 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:41.476449 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:41.597023 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:41.809468 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:41.936437 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:42.146605 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:42.213106 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:42.275380 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:42.317675 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:42.490471 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:42.619984 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:42.833870 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:42.944581 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240426 09:59:42.966394 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:43.093526 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 09:59:43.182124 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:43.222754 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:43.314899 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:43.358395 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:43.531172 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:43.535543 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:43.850924 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:43.890622 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:44.049440 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240426 09:59:44.095281 5254 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:44.095552 5254 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:44.096220 5254 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:44.097553 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:44.097868 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:44.098026 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:44.098273 5254 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:44.216552 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:44.228291 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:44.253293 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:44.397684 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:44.578125 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:44.618710 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:44.715765 5254 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:44.715919 5254 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:44.716557 5254 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:44.717696 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:44.718142 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:44.718302 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:44.718545 5254 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:59:44.944157 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:44.946048 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:45.171048 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:45.221262 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:45.317294 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:45.484771 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:45.547688 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:45.673291 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240426 09:59:45.692028 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:45.776422 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240426 09:59:45.926991 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:46.072717 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:46.234354 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:46.312448 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:46.418365 5254 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:46.418547 5254 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:46.419209 5254 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:46.420449 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:46.420725 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:46.420825 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:46.421176 5254 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 09:59:46.454097 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:46.506460 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:46.701707 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:46.842110 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:47.089607 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:47.218251 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:47.231918 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:47.487531 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:47.568586 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:47.629078 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:47.887609 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:48.028267 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:48.221554 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:48.229557 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 09:59:48.230868 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240426 09:59:48.290474 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:48.436239 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:48.570467 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:48.697923 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:48.847702 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:49.111784 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:49.113041 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20240426 09:59:49.213896 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:49.260145 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:49.523111 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:49.584159 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:49.679816 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:49.942781 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:49.968677 5261 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 09:59:49.968868 5261 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 09:59:49.969588 5261 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 09:59:49.970966 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:49.971222 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 09:59:49.971343 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 09:59:49.971632 5261 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 09:59:50.103495 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:50.211520 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:50.365815 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:50.526970 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:50.590219 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:50.706259 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240426 09:59:50.797660 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:50.824623 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240426 09:59:50.953761 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:51.223753 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:51.226955 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:51.238183 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:51.605885 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:51.653667 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:51.657124 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:52.046068 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:52.093427 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:52.218300 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:52.485965 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:52.538515 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:52.709254 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:52.930733 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:52.986306 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:53.216188 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240426 09:59:53.221751 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:53.359130 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 09:59:53.379642 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:53.436456 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:53.833663 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:53.846223 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:53.889679 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:54.227960 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:54.293138 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:54.345677 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:54.483654 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 09:59:54.758277 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:54.758277 4821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:54.987649 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:55.035984 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:55.223539 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:55.226744 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:55.475888 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240426 09:59:55.501950 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:55.697839 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:55.803876 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240426 09:59:55.973151 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:56.152340 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:56.171865 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:56.233122 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:56.451444 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:56.651499 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:56.930017 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:57.134685 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:57.172405 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 09:59:57.234774 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:57.416450 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:57.619383 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:57.901787 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:57.995114 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 09:59:58.108893 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:58.218542 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:58.304494 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 09:59:58.392765 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:58.422170 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240426 09:59:58.601583 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:58.892305 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:59.099606 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 09:59:59.218227 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:59:59.322556 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 09:59:59.397254 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:59.600414 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 09:59:59.646737 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240426 09:59:59.902153 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:00.108430 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:00.227378 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:00.325210 5261 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:00.325394 5261 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:00.325981 5261 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:00.327400 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:00.327600 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 10:00:00.327699 4794 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:00.328128 5261 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 10:00:00.329203 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240426 10:00:00.365060 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:00.413529 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:00.617856 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:00.926635 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:01.132401 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:01.159989 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240426 10:00:01.220039 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:01.389700 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:01.442301 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:01.646661 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:01.961962 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:02.169970 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:02.210974 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:02.434437 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:02.489728 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:02.525202 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240426 10:00:02.695996 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:03.018555 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:03.228473 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:03.231540 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:03.497784 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:03.550823 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:03.764609 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:03.846102 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240426 10:00:04.091974 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:04.213213 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:04.300932 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:04.572678 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:04.636379 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:04.845329 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:04.910236 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 10:00:05.084769 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240426 10:00:05.184901 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:05.222800 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:05.390898 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:05.667795 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:05.732532 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:05.943468 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:05.953258 5261 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:05.953433 5261 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:05.954042 5261 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:05.955502 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:05.955853 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:05.955971 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:05.956194 5261 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 10:00:06.234855 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:06.266511 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240426 10:00:06.283917 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:06.500126 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:06.778724 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:06.842646 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:07.060009 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:07.217288 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:07.403878 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:07.625169 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:07.667939 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240426 10:00:07.910071 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:07.970906 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:08.192920 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:08.225371 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:08.535130 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:08.564797 5261 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:08.564993 5261 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:08.565752 5261 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:08.567193 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:08.567339 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:08.567461 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:08.567672 5261 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 10:00:08.738533 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240426 10:00:08.761549 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:09.048597 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:09.111537 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:09.232367 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:09.337020 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:09.683758 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:09.916388 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:09.952170 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240426 10:00:09.952991 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240426 10:00:10.208703 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:10.220415 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:10.261755 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:10.263808 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:10.794605 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:10.844291 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:11.218917 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:11.372004 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240426 10:00:11.381019 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:11.427671 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 10:00:11.643651 5272 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:11.643905 5272 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:11.644735 5272 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:11.646306 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:11.646636 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 10:00:11.646768 4794 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:11.647054 5272 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 10:00:11.720826 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:11.973287 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:12.222503 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:12.312161 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:12.323338 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240426 10:00:12.568413 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:12.612134 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:12.908380 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:13.212594 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:13.213387 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:13.504751 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:13.774024 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:13.817426 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:13.951529 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240426 10:00:14.107777 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:14.223273 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:14.429503 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:14.663980 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240426 10:00:14.717589 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:14.777156 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:15.037433 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:15.089326 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240426 10:00:15.205425 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:15.327790 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:15.652395 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:15.919601 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:15.940515 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:16.092835 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240426 10:00:16.208770 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:16.269224 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:16.557879 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:16.896649 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:16.922489 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:16.992242 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240426 10:00:17.179627 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:17.219069 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:17.529388 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:17.807509 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:18.106185 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:18.161722 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:18.205365 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:18.322778 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240426 10:00:18.435685 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:18.799100 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:19.067061 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:19.118589 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
I20240426 10:00:19.204048 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:19.440755 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:19.587710 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240426 10:00:19.701334 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:20.084913 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:20.204846 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:20.341944 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:20.342967 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:20.416710 4794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240426 10:00:20.733819 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:20.901455 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240426 10:00:20.994632 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:21.208914 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:21.367666 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:21.387027 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:21.650082 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:21.814725 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240426 10:00:22.044158 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:22.200350 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:22.311810 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:22.372710 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:22.701501 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:22.976605 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:23.209662 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:23.287396 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240426 10:00:23.369529 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:23.606514 5040 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:23.646834 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:24.041343 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:24.186889 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:24.316406 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:24.377768 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240426 10:00:24.652089 5039 meta_cache.cc:302] tablet cd33b93d8d5a4fa98623df51a9b812e9: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:24.713107 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:24.832393 5276 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:24.832585 5276 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:24.833242 5276 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:24.834509 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:24.834930 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:24.835042 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:24.835404 5276 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:00:24.991065 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:25.203786 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:25.390086 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:25.418267 4795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 10:00:25.669173 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:25.672204 5040 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
W20240426 10:00:26.013051 4795 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Couldn't send request to peer bfe3ccee0ac048e6b426132b5a31254f. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240426 10:00:26.071942 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:26.224300 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:26.354916 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:26.691252 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:26.759788 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:26.775621 5276 raft_consensus.cc:482] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:26.775772 5276 raft_consensus.cc:504] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:26.776373 5276 leader_election.cc:288] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:26.777796 4794 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:26.777932 4795 leader_election.cc:334] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:26.778038 4795 leader_election.cc:302] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:26.778264 5276 raft_consensus.cc:2726] T cd33b93d8d5a4fa98623df51a9b812e9 P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 10:00:26.813217 4794 consensus_peers.cc:485] T eb13bec03b614847beab71dc877dccaf P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 -> Peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Couldn't send request to peer 563b22b809134189a9e7b5515501a6a3. Status: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240426 10:00:27.045091 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
I20240426 10:00:27.220350 5088 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:27.326251 5276 raft_consensus.cc:482] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 563b22b809134189a9e7b5515501a6a3)
I20240426 10:00:27.326434 5276 raft_consensus.cc:504] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b22b809134189a9e7b5515501a6a3" member_type: VOTER last_known_addr { host: "127.4.112.193" port: 46491 } } peers { permanent_uuid: "0e4d0ccb49a84cacb9ad7f3e2e6347a0" member_type: VOTER last_known_addr { host: "127.4.112.194" port: 46279 } } peers { permanent_uuid: "bfe3ccee0ac048e6b426132b5a31254f" member_type: VOTER last_known_addr { host: "127.4.112.195" port: 35147 } }
I20240426 10:00:27.327075 5276 leader_election.cc:288] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491), bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147)
W20240426 10:00:27.328296 4794 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491): Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:27.328431 4795 leader_election.cc:334] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfe3ccee0ac048e6b426132b5a31254f (127.4.112.195:35147): Network error: Client connection negotiation failed: client connection to 127.4.112.195:35147: connect: Connection refused (error 111)
I20240426 10:00:27.328534 4795 leader_election.cc:302] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4d0ccb49a84cacb9ad7f3e2e6347a0; no voters: 563b22b809134189a9e7b5515501a6a3, bfe3ccee0ac048e6b426132b5a31254f
I20240426 10:00:27.328758 5276 raft_consensus.cc:2726] T 60e1df189e6e4c9289246a204323b23a P 0e4d0ccb49a84cacb9ad7f3e2e6347a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:00:27.446492 4820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45686: Illegal state: replica 0e4d0ccb49a84cacb9ad7f3e2e6347a0 is not leader of this config: current role FOLLOWER
W20240426 10:00:27.715081 5039 meta_cache.cc:302] tablet 60e1df189e6e4c9289246a204323b23a: replica 563b22b809134189a9e7b5515501a6a3 (127.4.112.193:46491) has failed: Network error: Client connection negotiation failed: client connection to 127.4.112.193:46491: connect: Connection refused (error 111)
W20240426 10:00:27.732692 5039 batcher.cc:441] Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.733333 4547 client-test-util.cc:49] 33 failed ops. First 10 errors follow
I20240426 10:00:27.733491 4547 client-test-util.cc:55] Op INSERT int64 rand_key=1487340986486554625, int64 link_to=9217504936005926913, int64 insert_ts=1714125567628696 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.733747 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-6049047613650960382, int64 link_to=-7102984248709349374, int64 insert_ts=1714125567628760 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.733863 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-1188021228706463739, int64 link_to=6367311940261838853, int64 insert_ts=1714125567628925 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.733919 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-8724550566332268538, int64 link_to=8493848304232824838, int64 insert_ts=1714125567628979 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.733991 4547 client-test-util.cc:55] Op INSERT int64 rand_key=2185664169751478279, int64 link_to=-7826640880482451449, int64 insert_ts=1714125567629030 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.734053 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-3863524181387771895, int64 link_to=3517400419494461449, int64 insert_ts=1714125567629180 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.734134 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-489698045441540085, int64 link_to=7769910197483012107, int64 insert_ts=1714125567629270 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.734203 4547 client-test-util.cc:55] Op INSERT int64 rand_key=997642941045014540, int64 link_to=-1459047465243901940, int64 insert_ts=1714125567629323 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.734275 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-6538886396580790259, int64 link_to=667207423750373389, int64 insert_ts=1714125567629366 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
I20240426 10:00:27.734338 4547 client-test-util.cc:55] Op INSERT int64 rand_key=-3165060260634558449, int64 link_to=4919998676715634703, int64 insert_ts=1714125567629458 had status Timed out: Failed to write batch of 33 ops to tablet cd33b93d8d5a4fa98623df51a9b812e9 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: cd33b93d8d5a4fa98623df51a9b812e9, num_ops: 33, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.112.193:46491 (error 108)
F20240426 10:00:27.734428 4547 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714125627 (unix time) try "date -d @1714125627" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000011c3) received by PID 4547 (TID 0x7feb4fa38d80) from PID 4547; stack trace: ***
@ 0x7feb57440980 (unknown) at ??:0
@ 0x7feb504f0fb7 gsignal at ??:0
@ 0x7feb504f2921 abort at ??:0
@ 0x7feb53218dbd google::LogMessage::Fail() at ??:0
@ 0x7feb5321cb83 google::LogMessage::SendToLog() at ??:0
@ 0x7feb532187bc google::LogMessage::Flush() at ??:0
@ 0x7feb53219f49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feb62c9bebd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed282 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/client/client-test-util.h:?
@ 0x3d5f48 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8249 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7feb550b20ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7feb550a6bdb testing::Test::Run() at ??:0
@ 0x7feb550a6d9d testing::TestInfo::Run() at ??:0
@ 0x7feb550a7377 testing::TestSuite::Run() at ??:0
@ 0x7feb550a777c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7feb550b260d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7feb550a6e63 testing::UnitTest::Run() at ??:0
@ 0x7feb69d7a272 RUN_ALL_TESTS() at ??:0
@ 0x7feb69d78789 main at ??:0
@ 0x7feb504d3bf7 __libc_start_main at ??:0
@ 0x2eb91a _start at ??:?