[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 20:11:58.899248 29000 test_util.cc:268] Using random seed: -75991039
I20240430 20:11:58.911446 29000 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 20:11:58.911556 29000 linked_list-test.cc:101] --------------
I20240430 20:11:58.911621 29000 linked_list-test.cc:102] 50 chains
I20240430 20:11:58.911693 29000 linked_list-test.cc:103] 3 tablets
I20240430 20:11:58.911772 29000 linked_list-test.cc:104] Mutations on
I20240430 20:11:58.911823 29000 linked_list-test.cc:105] --------------
I20240430 20:11:58.912076 29000 ts_itest-base.cc:112] Starting cluster with:
I20240430 20:11:58.912145 29000 ts_itest-base.cc:113] --------------
I20240430 20:11:58.912209 29000 ts_itest-base.cc:114] 3 tablet servers
I20240430 20:11:58.912264 29000 ts_itest-base.cc:115] 3 replicas per TS
I20240430 20:11:58.912333 29000 ts_itest-base.cc:116] --------------
2024-04-30T20:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T20:11:58Z Disabled control of system clock
I20240430 20:11:58.938251 29000 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
/tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.82.62:34925
--webserver_interface=127.28.82.62
--webserver_port=0
--builtin_ntp_servers=127.28.82.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.82.62:34925
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 20:11:59.268386 29012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:59.268733 29012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:59.268884 29012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:59.276583 29012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:59.276724 29012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:59.276769 29012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:59.276866 29012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:59.286327 29012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.82.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.82.62:34925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.82.62:34925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.28.82.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:59.287744 29012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:59.289558 29012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:59.301615 29017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:59.302281 29020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:59.302731 29018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:59.303618 29012 server_base.cc:1034] running on GCE node
I20240430 20:11:59.304526 29012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:59.306131 29012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:59.307439 29012 hybrid_clock.cc:648] HybridClock initialized: now 1714507919307378 us; error 32 us; skew 500 ppm
I20240430 20:11:59.307778 29012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:59.315863 29012 webserver.cc:460] Webserver started at http://127.28.82.62:36991/ using document root <none> and password file <none>
I20240430 20:11:59.316686 29012 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:59.316857 29012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:59.317188 29012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:59.319968 29012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data/instance:
uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.320713 29012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal/instance:
uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.326180 29012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240430 20:11:59.330004 29026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:59.331334 29012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240430 20:11:59.331535 29012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data,/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal
uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.331733 29012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:59.372411 29012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:59.373358 29012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:59.373694 29012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:59.392585 29012 rpc_server.cc:307] RPC server started. Bound to: 127.28.82.62:34925
I20240430 20:11:59.392604 29077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.82.62:34925 every 8 connection(s)
I20240430 20:11:59.394013 29012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/data/info.pb
I20240430 20:11:59.397279 29000 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu as pid 29012
I20240430 20:11:59.397580 29000 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/master-0/wal/instance
I20240430 20:11:59.398524 29078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:59.409842 29078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: Bootstrap starting.
I20240430 20:11:59.412897 29078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:59.414273 29078 log.cc:824] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:59.417604 29078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: No bootstrap required, opened a new log
I20240430 20:11:59.425966 29078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } }
I20240430 20:11:59.426334 29078 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:59.426460 29078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34cc5b3ad554c3f8e1dbdbf5c10f8f9, State: Initialized, Role: FOLLOWER
I20240430 20:11:59.427248 29078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } }
I20240430 20:11:59.427520 29078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 20:11:59.427698 29078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 20:11:59.427872 29078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:59.430011 29078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } }
I20240430 20:11:59.430437 29078 leader_election.cc:302] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34cc5b3ad554c3f8e1dbdbf5c10f8f9; no voters:
I20240430 20:11:59.431017 29078 leader_election.cc:288] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 20:11:59.431196 29083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:59.431866 29083 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [term 1 LEADER]: Becoming Leader. State: Replica: d34cc5b3ad554c3f8e1dbdbf5c10f8f9, State: Running, Role: LEADER
I20240430 20:11:59.432435 29083 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } }
I20240430 20:11:59.433151 29078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:11:59.435796 29084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } } }
I20240430 20:11:59.435770 29085 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34cc5b3ad554c3f8e1dbdbf5c10f8f9. Latest consensus state: current_term: 1 leader_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34cc5b3ad554c3f8e1dbdbf5c10f8f9" member_type: VOTER last_known_addr { host: "127.28.82.62" port: 34925 } } }
I20240430 20:11:59.436118 29084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:59.436180 29085 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9 [sys.catalog]: This master's current role is: LEADER
W20240430 20:11:59.445367 29099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 20:11:59.445501 29099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 20:11:59.445624 29091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:59.447974 29091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:59.457235 29091 catalog_manager.cc:1347] Generated new cluster ID: 20496ab54e074406a94ab4cb88e39f63
I20240430 20:11:59.457370 29091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:59.487125 29091 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 20:11:59.488463 29091 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:59.504695 29091 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d34cc5b3ad554c3f8e1dbdbf5c10f8f9: Generated new TSK 0
I20240430 20:11:59.505975 29091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:59.513536 29000 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
/tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.82.1:0
--local_ip_for_outbound_sockets=127.28.82.1
--webserver_interface=127.28.82.1
--webserver_port=0
--tserver_master_addrs=127.28.82.62:34925
--builtin_ntp_servers=127.28.82.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 20:11:59.848156 29102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:59.848479 29102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:59.848608 29102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:59.856134 29102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:59.856294 29102 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:11:59.856333 29102 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:11:59.856387 29102 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:11:59.856468 29102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.82.1
I20240430 20:11:59.865710 29102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.82.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.82.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.28.82.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.28.82.62:34925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.82.1
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:59.867063 29102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:59.868995 29102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:59.882638 29110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:59.883356 29108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:59.883460 29107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:59.884148 29102 server_base.cc:1034] running on GCE node
I20240430 20:11:59.884903 29102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:59.886873 29102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:59.888149 29102 hybrid_clock.cc:648] HybridClock initialized: now 1714507919888055 us; error 76 us; skew 500 ppm
I20240430 20:11:59.888631 29102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:59.891192 29102 webserver.cc:460] Webserver started at http://127.28.82.1:39241/ using document root <none> and password file <none>
I20240430 20:11:59.891937 29102 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:59.892035 29102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:59.892308 29102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:59.895100 29102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data/instance:
uuid: "999f86490d934d6c9f2f438b6000f584"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.895857 29102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal/instance:
uuid: "999f86490d934d6c9f2f438b6000f584"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.901556 29102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240430 20:11:59.905172 29116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:59.906818 29102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240430 20:11:59.907008 29102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal
uuid: "999f86490d934d6c9f2f438b6000f584"
format_stamp: "Formatted at 2024-04-30 20:11:59 on dist-test-slave-sst2"
I20240430 20:11:59.907217 29102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:59.928802 29102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:59.929824 29102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:59.930140 29102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:59.931231 29102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:11:59.933073 29102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:11:59.933182 29102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:59.933277 29102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:11:59.933357 29102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:59.964897 29102 rpc_server.cc:307] RPC server started. Bound to: 127.28.82.1:43573
I20240430 20:11:59.964983 29228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.82.1:43573 every 8 connection(s)
I20240430 20:11:59.966251 29102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/data/info.pb
I20240430 20:11:59.971702 29000 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu as pid 29102
I20240430 20:11:59.971920 29000 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-0/wal/instance
I20240430 20:11:59.975473 29000 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
/tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.82.2:0
--local_ip_for_outbound_sockets=127.28.82.2
--webserver_interface=127.28.82.2
--webserver_port=0
--tserver_master_addrs=127.28.82.62:34925
--builtin_ntp_servers=127.28.82.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 20:11:59.980573 29229 heartbeater.cc:347] Connected to a master server at 127.28.82.62:34925
I20240430 20:11:59.980926 29229 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:59.981809 29229 heartbeater.cc:511] Master 127.28.82.62:34925 requested a full tablet report, sending...
I20240430 20:11:59.984015 29043 ts_manager.cc:194] Registered new tserver with Master: 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573)
I20240430 20:11:59.985610 29043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.82.1:41487
W20240430 20:12:00.307667 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:12:00.308020 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:12:00.308204 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:12:00.315951 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:12:00.316138 29233 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:12:00.316202 29233 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:12:00.316304 29233 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:12:00.316431 29233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.82.2
I20240430 20:12:00.325801 29233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.82.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.82.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.28.82.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.28.82.62:34925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.82.2
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:12:00.327196 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:12:00.329006 29233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:12:00.341818 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:12:00.342322 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:12:00.342873 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:12:00.343855 29233 server_base.cc:1034] running on GCE node
I20240430 20:12:00.344520 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:12:00.345572 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:12:00.346812 29233 hybrid_clock.cc:648] HybridClock initialized: now 1714507920346761 us; error 52 us; skew 500 ppm
I20240430 20:12:00.347179 29233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:12:00.349691 29233 webserver.cc:460] Webserver started at http://127.28.82.2:35829/ using document root <none> and password file <none>
I20240430 20:12:00.350474 29233 fs_manager.cc:359] Metadata directory not provided
I20240430 20:12:00.350592 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:12:00.350909 29233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:12:00.353696 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data/instance:
uuid: "142855f2169447a29ccb6ba6a9a7fe46"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.354460 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal/instance:
uuid: "142855f2169447a29ccb6ba6a9a7fe46"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.359724 29233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240430 20:12:00.363353 29247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.364753 29233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240430 20:12:00.364970 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal
uuid: "142855f2169447a29ccb6ba6a9a7fe46"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.365167 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:12:00.388305 29233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:12:00.389221 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:12:00.389518 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:12:00.390480 29233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:12:00.392045 29233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:12:00.392144 29233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.392263 29233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:12:00.392335 29233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.425504 29233 rpc_server.cc:307] RPC server started. Bound to: 127.28.82.2:36545
I20240430 20:12:00.425561 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.82.2:36545 every 8 connection(s)
I20240430 20:12:00.426803 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/data/info.pb
I20240430 20:12:00.434023 29000 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu as pid 29233
I20240430 20:12:00.434294 29000 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-1/wal/instance
I20240430 20:12:00.438328 29000 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
/tmp/dist-test-task2JZJ6D/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.82.3:0
--local_ip_for_outbound_sockets=127.28.82.3
--webserver_interface=127.28.82.3
--webserver_port=0
--tserver_master_addrs=127.28.82.62:34925
--builtin_ntp_servers=127.28.82.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 20:12:00.440508 29360 heartbeater.cc:347] Connected to a master server at 127.28.82.62:34925
I20240430 20:12:00.440824 29360 heartbeater.cc:464] Registering TS with master...
I20240430 20:12:00.441644 29360 heartbeater.cc:511] Master 127.28.82.62:34925 requested a full tablet report, sending...
I20240430 20:12:00.443195 29043 ts_manager.cc:194] Registered new tserver with Master: 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545)
I20240430 20:12:00.443987 29043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.82.2:38951
W20240430 20:12:00.777395 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:12:00.777700 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:12:00.777863 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:12:00.785377 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:12:00.785540 29363 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:12:00.785616 29363 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:12:00.785702 29363 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:12:00.785812 29363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.82.3
I20240430 20:12:00.794986 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.82.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.82.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.28.82.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.28.82.62:34925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.82.3
--log_dir=/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:12:00.796406 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:12:00.798241 29363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:12:00.813745 29369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:12:00.814100 29368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:12:00.814919 29371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:12:00.815806 29363 server_base.cc:1034] running on GCE node
I20240430 20:12:00.816465 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:12:00.817605 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:12:00.818853 29363 hybrid_clock.cc:648] HybridClock initialized: now 1714507920818798 us; error 52 us; skew 500 ppm
I20240430 20:12:00.819217 29363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:12:00.821754 29363 webserver.cc:460] Webserver started at http://127.28.82.3:34767/ using document root <none> and password file <none>
I20240430 20:12:00.822543 29363 fs_manager.cc:359] Metadata directory not provided
I20240430 20:12:00.822667 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:12:00.823005 29363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:12:00.826165 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data/instance:
uuid: "b3858d6e040645898f743997947970ad"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.828953 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal/instance:
uuid: "b3858d6e040645898f743997947970ad"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.834735 29363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240430 20:12:00.838827 29377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.840495 29363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240430 20:12:00.840695 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal
uuid: "b3858d6e040645898f743997947970ad"
format_stamp: "Formatted at 2024-04-30 20:12:00 on dist-test-slave-sst2"
I20240430 20:12:00.840965 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:12:00.867669 29363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:12:00.868659 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:12:00.869001 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:12:00.870137 29363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:12:00.871806 29363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:12:00.871914 29363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.872040 29363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:12:00.872115 29363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:12:00.909798 29363 rpc_server.cc:307] RPC server started. Bound to: 127.28.82.3:46029
I20240430 20:12:00.909847 29489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.82.3:46029 every 8 connection(s)
I20240430 20:12:00.911121 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/data/info.pb
I20240430 20:12:00.919236 29000 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2JZJ6D/build/asan/bin/kudu as pid 29363
I20240430 20:12:00.919467 29000 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2JZJ6D/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507918817261-29000-0/linked-list-cluster/ts-2/wal/instance
I20240430 20:12:00.925274 29490 heartbeater.cc:347] Connected to a master server at 127.28.82.62:34925
I20240430 20:12:00.925594 29490 heartbeater.cc:464] Registering TS with master...
I20240430 20:12:00.926435 29490 heartbeater.cc:511] Master 127.28.82.62:34925 requested a full tablet report, sending...
I20240430 20:12:00.927911 29043 ts_manager.cc:194] Registered new tserver with Master: b3858d6e040645898f743997947970ad (127.28.82.3:46029)
I20240430 20:12:00.928845 29043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.82.3:47145
I20240430 20:12:00.937201 29000 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:12:00.961176 29043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58370:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240430 20:12:00.968515 29043 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:12:00.988590 29229 heartbeater.cc:503] Master 127.28.82.62:34925 was elected leader, sending a full tablet report...
I20240430 20:12:01.003592 29295 tablet_service.cc:1467] Processing CreateTablet for tablet b8b94089cd004f6aab04ea4d067bdb82 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:12:01.005192 29294 tablet_service.cc:1467] Processing CreateTablet for tablet 18b59309bf554ce5a8d95e3a923f1b97 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:12:01.004387 29424 tablet_service.cc:1467] Processing CreateTablet for tablet 18b59309bf554ce5a8d95e3a923f1b97 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:12:01.004437 29423 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2e9cef5024ea8b1f1d5a145331641 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:12:01.006248 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b59309bf554ce5a8d95e3a923f1b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.006441 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2e9cef5024ea8b1f1d5a145331641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.004503 29425 tablet_service.cc:1467] Processing CreateTablet for tablet b8b94089cd004f6aab04ea4d067bdb82 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:12:01.007349 29425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b94089cd004f6aab04ea4d067bdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.008980 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b59309bf554ce5a8d95e3a923f1b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.011230 29293 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2e9cef5024ea8b1f1d5a145331641 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:12:01.011705 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2e9cef5024ea8b1f1d5a145331641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.013134 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 18b59309bf554ce5a8d95e3a923f1b97 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:12:01.015174 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b59309bf554ce5a8d95e3a923f1b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.015296 29295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b94089cd004f6aab04ea4d067bdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.016950 29164 tablet_service.cc:1467] Processing CreateTablet for tablet b8b94089cd004f6aab04ea4d067bdb82 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:12:01.017913 29164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b94089cd004f6aab04ea4d067bdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.017412 29162 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2e9cef5024ea8b1f1d5a145331641 (DEFAULT_TABLE table=TestTable [id=daa13add477249d29b2bcb2dfa25634a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:12:01.024938 29162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2e9cef5024ea8b1f1d5a145331641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:12:01.029982 29508 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: Bootstrap starting.
I20240430 20:12:01.031335 29509 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: Bootstrap starting.
I20240430 20:12:01.033736 29508 tablet_bootstrap.cc:654] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.035142 29508 log.cc:824] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: Log is configured to *not* fsync() on all Append() calls
I20240430 20:12:01.035470 29509 tablet_bootstrap.cc:654] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.037312 29509 log.cc:824] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: Log is configured to *not* fsync() on all Append() calls
I20240430 20:12:01.045094 29508 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: No bootstrap required, opened a new log
I20240430 20:12:01.045450 29508 ts_tablet_manager.cc:1367] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240430 20:12:01.046133 29512 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: Bootstrap starting.
I20240430 20:12:01.047430 29509 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: No bootstrap required, opened a new log
I20240430 20:12:01.047717 29509 ts_tablet_manager.cc:1367] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240430 20:12:01.050069 29512 tablet_bootstrap.cc:654] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.051108 29512 log.cc:824] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: Log is configured to *not* fsync() on all Append() calls
I20240430 20:12:01.053802 29512 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: No bootstrap required, opened a new log
I20240430 20:12:01.054090 29512 ts_tablet_manager.cc:1367] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240430 20:12:01.054872 29508 raft_consensus.cc:348] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.055335 29508 raft_consensus.cc:374] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.055455 29508 raft_consensus.cc:729] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3858d6e040645898f743997947970ad, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.056218 29508 consensus_queue.cc:260] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.056692 29509 raft_consensus.cc:348] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.057204 29509 raft_consensus.cc:374] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.057310 29509 raft_consensus.cc:729] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142855f2169447a29ccb6ba6a9a7fe46, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.057622 29490 heartbeater.cc:503] Master 127.28.82.62:34925 was elected leader, sending a full tablet report...
I20240430 20:12:01.058192 29508 ts_tablet_manager.cc:1398] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20240430 20:12:01.058132 29509 consensus_queue.cc:260] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.059281 29360 heartbeater.cc:503] Master 127.28.82.62:34925 was elected leader, sending a full tablet report...
I20240430 20:12:01.059958 29509 ts_tablet_manager.cc:1398] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20240430 20:12:01.060350 29509 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46: Bootstrap starting.
I20240430 20:12:01.061220 29508 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: Bootstrap starting.
I20240430 20:12:01.062644 29512 raft_consensus.cc:348] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.063407 29512 raft_consensus.cc:374] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.063565 29512 raft_consensus.cc:729] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999f86490d934d6c9f2f438b6000f584, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.063864 29509 tablet_bootstrap.cc:654] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.064006 29508 tablet_bootstrap.cc:654] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.064458 29512 consensus_queue.cc:260] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.075340 29509 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46: No bootstrap required, opened a new log
I20240430 20:12:01.075562 29509 ts_tablet_manager.cc:1367] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20240430 20:12:01.076206 29512 ts_tablet_manager.cc:1398] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584: Time spent starting tablet: real 0.022s user 0.009s sys 0.012s
I20240430 20:12:01.076442 29508 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: No bootstrap required, opened a new log
I20240430 20:12:01.076661 29508 ts_tablet_manager.cc:1367] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240430 20:12:01.076606 29509 raft_consensus.cc:348] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.076857 29512 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: Bootstrap starting.
I20240430 20:12:01.076870 29509 raft_consensus.cc:374] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.076961 29509 raft_consensus.cc:729] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142855f2169447a29ccb6ba6a9a7fe46, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.077224 29509 consensus_queue.cc:260] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.077962 29508 raft_consensus.cc:348] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.078238 29508 raft_consensus.cc:374] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.078378 29508 raft_consensus.cc:729] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3858d6e040645898f743997947970ad, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.078881 29512 tablet_bootstrap.cc:654] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.078718 29508 consensus_queue.cc:260] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.080148 29508 ts_tablet_manager.cc:1398] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:12:01.080440 29509 ts_tablet_manager.cc:1398] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240430 20:12:01.080626 29508 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad: Bootstrap starting.
I20240430 20:12:01.081569 29509 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: Bootstrap starting.
I20240430 20:12:01.083227 29508 tablet_bootstrap.cc:654] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.083288 29509 tablet_bootstrap.cc:654] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.087786 29512 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: No bootstrap required, opened a new log
I20240430 20:12:01.088009 29512 ts_tablet_manager.cc:1367] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240430 20:12:01.089426 29512 raft_consensus.cc:348] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.089737 29512 raft_consensus.cc:374] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.089852 29512 raft_consensus.cc:729] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999f86490d934d6c9f2f438b6000f584, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.090269 29512 consensus_queue.cc:260] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.091033 29512 ts_tablet_manager.cc:1398] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240430 20:12:01.091439 29509 tablet_bootstrap.cc:492] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: No bootstrap required, opened a new log
I20240430 20:12:01.091542 29512 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584: Bootstrap starting.
I20240430 20:12:01.091640 29509 ts_tablet_manager.cc:1367] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240430 20:12:01.092398 29509 raft_consensus.cc:348] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.092672 29509 raft_consensus.cc:374] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.092761 29509 raft_consensus.cc:729] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142855f2169447a29ccb6ba6a9a7fe46, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.093096 29509 consensus_queue.cc:260] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.093981 29512 tablet_bootstrap.cc:654] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584: Neither blocks nor log segments found. Creating new log.
I20240430 20:12:01.095757 29509 ts_tablet_manager.cc:1398] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240430 20:12:01.096400 29508 tablet_bootstrap.cc:492] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad: No bootstrap required, opened a new log
I20240430 20:12:01.096598 29508 ts_tablet_manager.cc:1367] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20240430 20:12:01.097215 29512 tablet_bootstrap.cc:492] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584: No bootstrap required, opened a new log
I20240430 20:12:01.097261 29508 raft_consensus.cc:348] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.097401 29512 ts_tablet_manager.cc:1367] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240430 20:12:01.097498 29508 raft_consensus.cc:374] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.097582 29508 raft_consensus.cc:729] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3858d6e040645898f743997947970ad, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.097869 29508 consensus_queue.cc:260] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.098055 29512 raft_consensus.cc:348] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.098289 29512 raft_consensus.cc:374] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:12:01.098362 29512 raft_consensus.cc:729] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999f86490d934d6c9f2f438b6000f584, State: Initialized, Role: FOLLOWER
I20240430 20:12:01.098464 29508 ts_tablet_manager.cc:1398] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:12:01.098613 29512 consensus_queue.cc:260] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.099192 29512 ts_tablet_manager.cc:1398] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:12:01.116513 29515 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:12:01.116799 29515 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.119176 29515 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
I20240430 20:12:01.126291 29445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b59309bf554ce5a8d95e3a923f1b97" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3858d6e040645898f743997947970ad" is_pre_election: true
I20240430 20:12:01.126292 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b59309bf554ce5a8d95e3a923f1b97" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584" is_pre_election: true
I20240430 20:12:01.126785 29184 raft_consensus.cc:2454] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 0.
I20240430 20:12:01.126788 29445 raft_consensus.cc:2454] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 0.
I20240430 20:12:01.127398 29251 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, 999f86490d934d6c9f2f438b6000f584; no voters:
I20240430 20:12:01.127789 29515 raft_consensus.cc:2781] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:12:01.127945 29515 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:12:01.128055 29515 raft_consensus.cc:3037] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.130257 29515 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.130910 29515 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 election: Requested vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
I20240430 20:12:01.131337 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b59309bf554ce5a8d95e3a923f1b97" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584"
I20240430 20:12:01.131439 29445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b59309bf554ce5a8d95e3a923f1b97" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3858d6e040645898f743997947970ad"
I20240430 20:12:01.131621 29184 raft_consensus.cc:3037] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.131667 29445 raft_consensus.cc:3037] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.133913 29445 raft_consensus.cc:2454] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 1.
I20240430 20:12:01.133904 29184 raft_consensus.cc:2454] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 1.
I20240430 20:12:01.134337 29249 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad; no voters:
I20240430 20:12:01.134667 29515 raft_consensus.cc:2781] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:12:01.135121 29515 raft_consensus.cc:686] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 LEADER]: Becoming Leader. State: Replica: 142855f2169447a29ccb6ba6a9a7fe46, State: Running, Role: LEADER
I20240430 20:12:01.135617 29515 consensus_queue.cc:237] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.140539 29043 catalog_manager.cc:5484] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2). New cstate: current_term: 1 leader_uuid: "142855f2169447a29ccb6ba6a9a7fe46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } health_report { overall_health: UNKNOWN } } }
W20240430 20:12:01.163044 29491 tablet.cc:2369] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 20:12:01.179113 29361 tablet.cc:2369] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 20:12:01.218799 29230 tablet.cc:2369] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:12:01.254021 29515 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:12:01.254280 29515 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.254963 29515 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
I20240430 20:12:01.255501 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2e9cef5024ea8b1f1d5a145331641" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584" is_pre_election: true
I20240430 20:12:01.255609 29445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2e9cef5024ea8b1f1d5a145331641" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3858d6e040645898f743997947970ad" is_pre_election: true
I20240430 20:12:01.255745 29184 raft_consensus.cc:2454] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 0.
I20240430 20:12:01.255921 29445 raft_consensus.cc:2454] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 0.
I20240430 20:12:01.256181 29251 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, 999f86490d934d6c9f2f438b6000f584; no voters:
I20240430 20:12:01.256454 29515 raft_consensus.cc:2781] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:12:01.256580 29515 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:12:01.256659 29515 raft_consensus.cc:3037] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.258502 29515 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.259059 29515 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 election: Requested vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
I20240430 20:12:01.259538 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2e9cef5024ea8b1f1d5a145331641" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584"
I20240430 20:12:01.259557 29445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2e9cef5024ea8b1f1d5a145331641" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3858d6e040645898f743997947970ad"
I20240430 20:12:01.259752 29445 raft_consensus.cc:3037] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.259752 29184 raft_consensus.cc:3037] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.261703 29445 raft_consensus.cc:2454] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 1.
I20240430 20:12:01.261710 29184 raft_consensus.cc:2454] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142855f2169447a29ccb6ba6a9a7fe46 in term 1.
I20240430 20:12:01.262183 29251 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, 999f86490d934d6c9f2f438b6000f584, b3858d6e040645898f743997947970ad; no voters:
I20240430 20:12:01.262485 29515 raft_consensus.cc:2781] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:12:01.262658 29515 raft_consensus.cc:686] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 LEADER]: Becoming Leader. State: Replica: 142855f2169447a29ccb6ba6a9a7fe46, State: Running, Role: LEADER
I20240430 20:12:01.262908 29515 consensus_queue.cc:237] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:01.265782 29043 catalog_manager.cc:5484] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2). New cstate: current_term: 1 leader_uuid: "142855f2169447a29ccb6ba6a9a7fe46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } health_report { overall_health: UNKNOWN } } }
I20240430 20:12:01.280613 29514 raft_consensus.cc:482] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:12:01.280854 29514 raft_consensus.cc:504] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.282765 29514 leader_election.cc:288] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545)
I20240430 20:12:01.289474 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "b3858d6e040645898f743997947970ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584" is_pre_election: true
I20240430 20:12:01.289736 29184 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3858d6e040645898f743997947970ad in term 0.
I20240430 20:12:01.290069 29315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "b3858d6e040645898f743997947970ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142855f2169447a29ccb6ba6a9a7fe46" is_pre_election: true
I20240430 20:12:01.290282 29381 leader_election.cc:302] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584, b3858d6e040645898f743997947970ad; no voters:
I20240430 20:12:01.290551 29315 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3858d6e040645898f743997947970ad in term 0.
I20240430 20:12:01.290794 29514 raft_consensus.cc:2781] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:12:01.290946 29514 raft_consensus.cc:482] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:12:01.291060 29514 raft_consensus.cc:3037] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.293107 29514 raft_consensus.cc:504] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.293728 29514 leader_election.cc:288] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [CANDIDATE]: Term 1 election: Requested vote from peers 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573), 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545)
I20240430 20:12:01.294353 29184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "b3858d6e040645898f743997947970ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584"
I20240430 20:12:01.294351 29315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "b3858d6e040645898f743997947970ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142855f2169447a29ccb6ba6a9a7fe46"
I20240430 20:12:01.294584 29184 raft_consensus.cc:3037] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.294584 29315 raft_consensus.cc:3037] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:12:01.297608 29184 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3858d6e040645898f743997947970ad in term 1.
I20240430 20:12:01.297608 29315 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3858d6e040645898f743997947970ad in term 1.
I20240430 20:12:01.298214 29378 leader_election.cc:302] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad; no voters:
I20240430 20:12:01.298584 29514 raft_consensus.cc:2781] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:12:01.299127 29514 raft_consensus.cc:686] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [term 1 LEADER]: Becoming Leader. State: Replica: b3858d6e040645898f743997947970ad, State: Running, Role: LEADER
I20240430 20:12:01.299607 29514 consensus_queue.cc:237] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:01.304528 29043 catalog_manager.cc:5484] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad reported cstate change: term changed from 0 to 1, leader changed from <none> to b3858d6e040645898f743997947970ad (127.28.82.3). New cstate: current_term: 1 leader_uuid: "b3858d6e040645898f743997947970ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20240430 20:12:01.357465 29000 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:12:01.360397 29000 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 999f86490d934d6c9f2f438b6000f584 to finish bootstrapping
I20240430 20:12:01.370349 29000 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 142855f2169447a29ccb6ba6a9a7fe46 to finish bootstrapping
I20240430 20:12:01.378068 29000 ts_itest-base.cc:233] Waiting for 3 tablets on tserver b3858d6e040645898f743997947970ad to finish bootstrapping
I20240430 20:12:01.386688 29532 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.28.82.2:35829/threadz
http://127.28.82.62:36991/ipki-ca-cert-pem
http://127.28.82.62:36991/tables
http://127.28.82.1:39241/tablets
http://127.28.82.62:36991/ipki-ca-cert
http://127.28.82.62:36991/mem-trackers
http://127.28.82.3:34767/mem-trackers
http://127.28.82.62:36991/metrics
http://127.28.82.2:35829/transactions?tablet_id=b8b2e9cef5024ea8b1f1d5a145331641
http://127.28.82.62:36991/dump-entities
http://127.28.82.2:35829/startup
http://127.28.82.3:34767/threadz
http://127.28.82.3:34767/stacks
http://127.28.82.1:39241/threadz
http://127.28.82.62:36991/stacks
http://127.28.82.1:39241/metrics
http://127.28.82.3:34767/threadz?group=all
http://127.28.82.62:36991/tablet-servers
http://127.28.82.3:34767/maintenance-manager
http://127.28.82.2:35829/mem-trackers
http://127.28.82.3:34767/scans
http://127.28.82.2:35829/maintenance-manager
http://127.28.82.1:39241/stacks
http://127.28.82.3:34767/startup
http://127.28.82.3:34767/transactions?tablet_id=b8b2e9cef5024ea8b1f1d5a145331641
http://127.28.82.62:36991/ipki-ca-cert-der
http://127.28.82.2:35829/metrics
http://127.28.82.1:39241/scans
http://127.28.82.3:34767/tablets
http://127.28.82.2:35829/tablets
http://127.28.82.62:36991/masters
http://127.28.82.1:39241/threadz?group=all
http://127.28.82.1:39241/transactions?tablet_id=b8b2e9cef5024ea8b1f1d5a145331641
http://127.28.82.2:35829/scans
http://127.28.82.62:36991/threadz
http://127.28.82.3:34767/metrics
http://127.28.82.1:39241/maintenance-manager
http://127.28.82.62:36991/threadz?group=all
http://127.28.82.2:35829/threadz?group=all
http://127.28.82.1:39241/startup
http://127.28.82.1:39241/mem-trackers
http://127.28.82.62:36991/startup
http://127.28.82.2:35829/stacks
I20240430 20:12:01.392982 29000 linked_list-test-util.h:401] Running for: 30.000s
I20240430 20:12:01.393144 29000 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 20:12:01.393257 29000 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240430 20:12:01.430841 29315 raft_consensus.cc:1261] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Refusing update from remote peer b3858d6e040645898f743997947970ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.430841 29184 raft_consensus.cc:1261] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Refusing update from remote peer b3858d6e040645898f743997947970ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.432889 29514 consensus_queue.cc:1035] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:12:01.435164 29514 consensus_queue.cc:1035] T b8b94089cd004f6aab04ea4d067bdb82 P b3858d6e040645898f743997947970ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:12:01.445348 29183 raft_consensus.cc:1261] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Refusing update from remote peer 142855f2169447a29ccb6ba6a9a7fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.446146 29525 consensus_queue.cc:1035] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:12:01.447161 29445 raft_consensus.cc:1261] T b8b2e9cef5024ea8b1f1d5a145331641 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Refusing update from remote peer 142855f2169447a29ccb6ba6a9a7fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.448390 29525 consensus_queue.cc:1035] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:12:01.449667 29444 raft_consensus.cc:1261] T 18b59309bf554ce5a8d95e3a923f1b97 P b3858d6e040645898f743997947970ad [term 1 FOLLOWER]: Refusing update from remote peer 142855f2169447a29ccb6ba6a9a7fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.450596 29182 raft_consensus.cc:1261] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Refusing update from remote peer 142855f2169447a29ccb6ba6a9a7fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:12:01.456347 29525 consensus_queue.cc:1035] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240430 20:12:01.464900 29515 consensus_queue.cc:1035] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240430 20:12:01.468940 29541 mvcc.cc:205] Tried to move back new op lower bound from 7022624446232821760 to 7022624445494931456. Current Snapshot: MvccSnapshot[applied={T|T < 7022624446232821760}]
I20240430 20:12:01.483817 29540 mvcc.cc:205] Tried to move back new op lower bound from 7022624446169567232 to 7022624445647310848. Current Snapshot: MvccSnapshot[applied={T|T < 7022624446169567232}]
I20240430 20:12:01.528110 29548 mvcc.cc:205] Tried to move back new op lower bound from 7022624446231265280 to 7022624444975816704. Current Snapshot: MvccSnapshot[applied={T|T < 7022624446231265280}]
I20240430 20:12:03.377422 29539 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240430 20:12:04.203256 29539 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f34b7be2dcd google::LogMessage::Fail() at ??:0
@ 0x7f34b7be6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f34b7be27cc google::LogMessage::Flush() at ??:0
@ 0x7f34b7be3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f34ba80bc80 kudu::operator-() at ??:0
@ 0x7f34c748c14a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f34c74565de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f34c7456336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f34c74032d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f34c9ed0d87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f34c9edf93c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f34c9edf7a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f34c391502a std::function<>::operator()() at ??:0
@ 0x7f34c390d7e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f34c39087f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f34c3906927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f34c3944089 handle_request at ??:0
@ 0x7f34c394676b worker_thread at ??:0
@ 0x7f34bdc576db start_thread at ??:0
@ 0x7f34b6a9571f clone at ??:0
W20240430 20:12:04.470117 29494 meta_cache.cc:302] tablet b8b94089cd004f6aab04ea4d067bdb82: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: recv got EOF from 127.28.82.3:46029 (error 108)
W20240430 20:12:04.481534 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.487306 29273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
I20240430 20:12:04.498924 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:04.505010 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.513154 29273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
I20240430 20:12:04.526212 29533 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:04.535800 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.550362 29273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.579874 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.584028 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.601877 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.604437 29249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:04.606295 29249 consensus_peers.cc:485] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:12:04.606303 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.606494 29249 consensus_peers.cc:485] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:12:04.619895 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.629216 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.641397 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.652366 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.657455 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.665814 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.693701 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.695621 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.710703 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.717120 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.746867 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.766453 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.767917 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.794027 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.806859 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.825800 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.844636 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.870755 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.871227 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.896631 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.931695 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.950625 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.959829 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:04.975836 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.021479 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.035670 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.057600 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.063890 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.125721 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.131930 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.160533 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.170828 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.229108 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.251855 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.264509 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.290132 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.342608 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.375137 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.379375 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.415984 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.458824 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.502882 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.504699 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.546761 29494 meta_cache.cc:302] tablet b8b94089cd004f6aab04ea4d067bdb82: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:05.548803 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.589167 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.633881 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.642884 29275 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57590: Illegal state: replica 142855f2169447a29ccb6ba6a9a7fe46 is not leader of this config: current role FOLLOWER
I20240430 20:12:05.690320 29602 raft_consensus.cc:482] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3858d6e040645898f743997947970ad)
I20240430 20:12:05.690523 29602 raft_consensus.cc:504] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:05.691308 29602 leader_election.cc:288] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3858d6e040645898f743997947970ad (127.28.82.3:46029), 999f86490d934d6c9f2f438b6000f584 (127.28.82.1:43573)
I20240430 20:12:05.691977 29182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "142855f2169447a29ccb6ba6a9a7fe46" candidate_term: 2 candidate_status { last_received { term: 1 index: 116 } } ignore_live_leader: false dest_uuid: "999f86490d934d6c9f2f438b6000f584" is_pre_election: true
W20240430 20:12:05.692745 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:05.693222 29249 leader_election.cc:334] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:05.693387 29249 leader_election.cc:302] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46; no voters: 999f86490d934d6c9f2f438b6000f584, b3858d6e040645898f743997947970ad
I20240430 20:12:05.693709 29602 raft_consensus.cc:2726] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:05.706238 29605 raft_consensus.cc:482] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3858d6e040645898f743997947970ad)
I20240430 20:12:05.706490 29605 raft_consensus.cc:504] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:05.708407 29605 leader_election.cc:288] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3858d6e040645898f743997947970ad (127.28.82.3:46029), 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545)
W20240430 20:12:05.710875 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:05.713071 29118 leader_election.cc:334] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:05.717363 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "999f86490d934d6c9f2f438b6000f584" candidate_term: 2 candidate_status { last_received { term: 1 index: 116 } } ignore_live_leader: false dest_uuid: "142855f2169447a29ccb6ba6a9a7fe46" is_pre_election: true
I20240430 20:12:05.717655 29314 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999f86490d934d6c9f2f438b6000f584 in term 1.
I20240430 20:12:05.718248 29117 leader_election.cc:302] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, 999f86490d934d6c9f2f438b6000f584; no voters: b3858d6e040645898f743997947970ad
I20240430 20:12:05.718750 29605 raft_consensus.cc:2781] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:12:05.718883 29605 raft_consensus.cc:482] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b3858d6e040645898f743997947970ad)
I20240430 20:12:05.719005 29605 raft_consensus.cc:3037] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:12:05.721172 29605 raft_consensus.cc:504] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:05.721818 29605 leader_election.cc:288] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 election: Requested vote from peers b3858d6e040645898f743997947970ad (127.28.82.3:46029), 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545)
I20240430 20:12:05.722440 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b94089cd004f6aab04ea4d067bdb82" candidate_uuid: "999f86490d934d6c9f2f438b6000f584" candidate_term: 2 candidate_status { last_received { term: 1 index: 116 } } ignore_live_leader: false dest_uuid: "142855f2169447a29ccb6ba6a9a7fe46"
I20240430 20:12:05.722704 29314 raft_consensus.cc:3037] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:12:05.723418 29118 leader_election.cc:334] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:05.726290 29314 raft_consensus.cc:2454] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999f86490d934d6c9f2f438b6000f584 in term 2.
I20240430 20:12:05.726809 29117 leader_election.cc:302] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 142855f2169447a29ccb6ba6a9a7fe46, 999f86490d934d6c9f2f438b6000f584; no voters: b3858d6e040645898f743997947970ad
I20240430 20:12:05.727231 29605 raft_consensus.cc:2781] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 2 FOLLOWER]: Leader election won for term 2
W20240430 20:12:05.727555 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Service unavailable: leader is not yet ready
I20240430 20:12:05.727802 29605 raft_consensus.cc:686] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [term 2 LEADER]: Becoming Leader. State: Replica: 999f86490d934d6c9f2f438b6000f584, State: Running, Role: LEADER
I20240430 20:12:05.728631 29605 consensus_queue.cc:237] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.116, Last appended by leader: 116, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } }
I20240430 20:12:05.732285 29039 catalog_manager.cc:5484] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 reported cstate change: term changed from 1 to 2, leader changed from b3858d6e040645898f743997947970ad (127.28.82.3) to 999f86490d934d6c9f2f438b6000f584 (127.28.82.1). New cstate: current_term: 2 leader_uuid: "999f86490d934d6c9f2f438b6000f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20240430 20:12:05.749570 29314 raft_consensus.cc:1261] T b8b94089cd004f6aab04ea4d067bdb82 P 142855f2169447a29ccb6ba6a9a7fe46 [term 2 FOLLOWER]: Refusing update from remote peer 999f86490d934d6c9f2f438b6000f584: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 2 index: 118. (index mismatch)
I20240430 20:12:05.750401 29610 consensus_queue.cc:1035] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
W20240430 20:12:05.750864 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 20:12:06.282507 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:12:06.320282 29533 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:07.238673 29249 consensus_peers.cc:485] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 20:12:07.345847 29249 consensus_peers.cc:485] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:12:07.944240 29361 maintenance_manager.cc:419] P 142855f2169447a29ccb6ba6a9a7fe46: Scheduling FlushMRSOp(b8b94089cd004f6aab04ea4d067bdb82): perf score=1.000000
I20240430 20:12:07.984916 29230 maintenance_manager.cc:419] P 999f86490d934d6c9f2f438b6000f584: Scheduling FlushMRSOp(b8b94089cd004f6aab04ea4d067bdb82): perf score=1.000000
W20240430 20:12:08.013106 29118 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.82.3:46029, user_credentials={real_user=slave}} blocked reactor thread for 64281.4us
W20240430 20:12:08.512706 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:12:09.241957 29252 maintenance_manager.cc:643] P 142855f2169447a29ccb6ba6a9a7fe46: FlushMRSOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 1.297s user 0.609s sys 0.080s Metrics: {"bytes_written":1028,"cfile_init":1,"compiler_manager_pool.queue_time_us":79756,"compiler_manager_pool.run_cpu_time_us":677536,"compiler_manager_pool.run_wall_time_us":1156280,"delete_count":0,"dirs.queue_time_us":8208,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":5903,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2503,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7896,"spinlock_wait_cycles":651648,"thread_start_us":13009,"threads_started":2,"update_count":134}
I20240430 20:12:09.244061 29361 maintenance_manager.cc:419] P 142855f2169447a29ccb6ba6a9a7fe46: Scheduling MajorDeltaCompactionOp(b8b94089cd004f6aab04ea4d067bdb82): perf score=0.008243
I20240430 20:12:09.330518 29252 maintenance_manager.cc:643] P 142855f2169447a29ccb6ba6a9a7fe46: MajorDeltaCompactionOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 0.086s user 0.061s sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1431,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1249,"dirs.run_cpu_time_us":328,"dirs.run_wall_time_us":2111,"lbm_read_time_us":1494,"lbm_reads_lt_1ms":24,"lbm_write_time_us":331,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":6399,"threads_started":1,"update_count":134}
I20240430 20:12:09.331554 29361 maintenance_manager.cc:419] P 142855f2169447a29ccb6ba6a9a7fe46: Scheduling UndoDeltaBlockGCOp(b8b94089cd004f6aab04ea4d067bdb82): 140284 bytes on disk
I20240430 20:12:09.338632 29252 maintenance_manager.cc:643] P 142855f2169447a29ccb6ba6a9a7fe46: UndoDeltaBlockGCOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240430 20:12:09.367471 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:12:09.437422 29533 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:12:09.537655 29121 maintenance_manager.cc:643] P 999f86490d934d6c9f2f438b6000f584: FlushMRSOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 1.552s user 0.544s sys 0.116s Metrics: {"bytes_written":1028,"cfile_init":1,"compiler_manager_pool.queue_time_us":17777,"delete_count":0,"dirs.queue_time_us":23603,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":16285,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2444,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7896,"thread_start_us":42257,"threads_started":2,"update_count":134}
I20240430 20:12:09.540014 29230 maintenance_manager.cc:419] P 999f86490d934d6c9f2f438b6000f584: Scheduling UndoDeltaBlockGCOp(b8b94089cd004f6aab04ea4d067bdb82): 140284 bytes on disk
I20240430 20:12:09.541499 29121 maintenance_manager.cc:643] P 999f86490d934d6c9f2f438b6000f584: UndoDeltaBlockGCOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240430 20:12:09.542613 29230 maintenance_manager.cc:419] P 999f86490d934d6c9f2f438b6000f584: Scheduling MajorDeltaCompactionOp(b8b94089cd004f6aab04ea4d067bdb82): perf score=0.008243
I20240430 20:12:09.650310 29121 maintenance_manager.cc:643] P 999f86490d934d6c9f2f438b6000f584: MajorDeltaCompactionOp(b8b94089cd004f6aab04ea4d067bdb82) complete. Timing: real 0.107s user 0.064s sys 0.015s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1431,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":15544,"dirs.run_cpu_time_us":332,"dirs.run_wall_time_us":5730,"lbm_read_time_us":436,"lbm_reads_lt_1ms":20,"lbm_write_time_us":323,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":17153,"threads_started":2,"update_count":134}
W20240430 20:12:09.869668 29249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 20:12:09.879390 29249 consensus_peers.cc:485] T b8b2e9cef5024ea8b1f1d5a145331641 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 20:12:09.939386 29249 consensus_peers.cc:485] T 18b59309bf554ce5a8d95e3a923f1b97 P 142855f2169447a29ccb6ba6a9a7fe46 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
F20240430 20:12:10.382424 29533 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f035f0efdcd google::LogMessage::Fail() at ??:0
@ 0x7f035f0f3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f035f0ef7cc google::LogMessage::Flush() at ??:0
@ 0x7f035f0f0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0361d18c80 kudu::operator-() at ??:0
@ 0x7f036e99914a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f036e9635de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f036e963336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f036e9102d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f03713ddd87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f03713ec93c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f03713ec7a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f036ae2202a std::function<>::operator()() at ??:0
@ 0x7f036ae1a7e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f036ae157f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f036ae13927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f036ae51089 handle_request at ??:0
@ 0x7f036ae5376b worker_thread at ??:0
@ 0x7f03651646db start_thread at ??:0
@ 0x7f035dfa271f clone at ??:0
W20240430 20:12:10.655558 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
W20240430 20:12:10.660456 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.679730 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.710294 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.745955 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.795982 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.855217 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.911988 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240430 20:12:10.912892 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 20:12:10.920336 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.924410 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.929359 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.937983 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:12:10.941028 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.958762 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:10.986227 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.002092 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.002092 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.059141 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.064325 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.073132 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.142283 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.152350 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.216806 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.233841 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.254364 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.290825 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.354228 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.359831 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.436125 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.481848 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.550266 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.607712 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.662603 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.673725 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:11.680763 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:11.746912 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.783921 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:11.871685 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:11.877408 29700 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:11.877607 29700 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:11.878401 29700 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:11.880153 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:11.880342 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:11.880537 29117 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:11.880923 29700 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:11.888526 29700 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:11.888754 29700 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:11.889578 29700 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:11.891472 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:11.891923 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:11.892066 29117 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:11.892321 29700 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:11.943382 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.012486 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.102559 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.160367 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.262219 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.267915 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.381929 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.419493 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.477880 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.567713 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.648839 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.699007 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:12.730103 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:12.761162 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.827111 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:12.953838 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.029804 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.150862 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.242532 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.358249 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.468232 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.516511 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 20:12:13.536480 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 20:12:13.578193 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:13.587031 29700 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:13.587323 29700 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:13.588336 29700 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:13.589991 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:13.590276 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:13.590464 29118 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:13.590847 29700 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:13.698741 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.708771 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:13.729660 29700 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:13.729879 29700 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:13.730631 29700 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:13.732057 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:13.732255 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:13.732414 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:13.732678 29700 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:13.787834 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:13.805344 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:13.938634 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.037835 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.185513 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.284960 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.442720 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.535975 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.761062 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:14.796327 29143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:14.797179 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:14.818739 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:14.987391 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.153347 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.272524 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.434280 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.570444 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.725464 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:15.779505 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:15.846128 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:15.879141 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:16.005921 29117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240430 20:12:16.006945 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 20:12:16.023551 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:16.135072 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 20:12:16.191823 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:16.337245 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:16.403463 29705 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:16.403671 29705 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:16.404424 29705 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:16.407155 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:16.407717 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:16.407832 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:16.408084 29705 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:16.478008 29705 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:16.478276 29705 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:16.479316 29705 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:16.480808 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:16.481041 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:16.481130 29117 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:16.481377 29705 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:16.513342 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:16.660691 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:16.831199 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:16.842305 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:16.882073 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:16.987900 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:17.178814 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:17.326030 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:17.529469 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:17.670157 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:17.840938 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:17.886237 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:17.907840 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:18.024632 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:18.250845 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:18.299284 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 20:12:18.383836 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:18.545905 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 20:12:18.624859 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:18.696583 29705 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:18.696810 29705 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:18.697497 29705 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:18.699141 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:18.699682 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:18.699787 29117 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:18.700026 29705 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:18.759691 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:18.883023 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:18.950598 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:19.009414 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:19.143877 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:19.400398 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:19.433910 29710 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:19.434131 29710 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:19.434926 29710 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:19.436839 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:19.437044 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:19.437228 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:19.437534 29710 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:19.665822 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:19.803709 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:19.932922 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:19.973060 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:20.065744 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:20.215394 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:20.476846 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:20.485042 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 20:12:20.633757 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:20.897655 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:20.901185 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 20:12:20.942721 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:21.011909 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:21.059638 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:21.289801 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240430 20:12:21.327565 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:21.413851 29710 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:21.414054 29710 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:21.414693 29710 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:21.416469 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:21.416601 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:21.416741 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:21.417059 29710 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:21.499104 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:21.704509 29710 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:21.704818 29710 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:21.705829 29710 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:21.707320 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:21.708566 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:21.708693 29118 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:21.709002 29710 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:21.763288 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:21.945114 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:21.957190 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:22.064625 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:22.208575 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:22.401778 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:22.663151 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:22.866050 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:22.913403 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 20:12:22.972633 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:23.086297 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:23.126408 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:23.216569 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240430 20:12:23.336817 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:23.602456 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:23.814896 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:23.852377 29710 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:23.852593 29710 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:23.853371 29710 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:23.854770 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:23.855207 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:23.855360 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:23.855737 29710 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:23.978482 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:24.082729 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:24.128667 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:24.307130 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:24.574358 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:24.809067 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:24.979745 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:25.071767 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:25.159974 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:25.324962 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:25.440199 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 20:12:25.578470 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:25.834723 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 20:12:25.844907 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:26.018436 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:26.093608 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:26.204416 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:26.373128 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:26.388612 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240430 20:12:26.625943 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:26.907850 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:27.035125 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:27.166728 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:27.247977 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:27.453161 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:27.713907 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:28.006472 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:28.054662 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240430 20:12:28.081128 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:28.266981 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:28.294845 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:28.566556 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:28.599314 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240430 20:12:28.826843 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:29.138525 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:29.205920 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:29.331168 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:29.398957 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:29.720220 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:29.979048 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:30.229465 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:30.311018 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:30.358592 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:30.446650 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240430 20:12:30.453588 29718 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:30.453785 29718 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:30.454533 29718 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:30.455878 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:30.456393 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:30.456570 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:30.456902 29718 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:30.572896 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:30.907452 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:31.104203 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 20:12:31.175931 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:31.317628 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:31.395942 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:31.413782 29117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 20:12:31.522979 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:31.786062 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:31.949795 29718 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:31.950014 29718 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:31.950747 29718 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:31.952509 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:31.952673 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:31.952831 29117 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:31.953105 29718 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:32.141759 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:32.349720 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:32.401880 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:32.443948 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:32.768558 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:32.856240 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 20:12:32.929561 29722 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:32.929778 29722 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:32.930615 29722 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:32.932379 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:32.932626 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:32.932828 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:32.933176 29718 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:33.031461 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:33.394542 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 20:12:33.406889 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:33.457269 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:33.480644 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:33.669615 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:34.053719 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:34.315639 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:34.486330 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:12:34.526953 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:34.704402 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:34.972820 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:35.290437 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 20:12:35.370776 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:35.559190 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:35.591285 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:35.633826 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:35.812547 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 20:12:36.038775 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:36.305398 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:36.610606 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:36.720144 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:36.757869 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:36.822928 29117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240430 20:12:36.984513 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:37.408684 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:37.617697 29728 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:37.617959 29728 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:37.618897 29728 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:37.620502 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:37.620651 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:37.620792 29118 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:37.621042 29728 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:37.667810 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:37.673561 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:37.734542 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 20:12:37.873814 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:38.106961 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:38.272150 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240430 20:12:38.372164 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:38.694413 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:38.814396 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:38.883749 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:39.076318 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:39.532307 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:39.730371 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:39.793776 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:40.017509 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:40.258877 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:40.426956 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 20:12:40.517833 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240430 20:12:40.524760 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:40.768828 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:40.990501 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:41.049172 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:41.259934 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:41.381263 29728 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:41.381464 29728 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:41.382120 29728 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:41.383467 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:41.383903 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:41.384056 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:41.384338 29728 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:41.737452 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:41.807734 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:42.001132 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:42.044246 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240430 20:12:42.239061 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:42.488986 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:42.754463 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:42.856261 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:42.938463 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 20:12:43.013160 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240430 20:12:43.252308 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:43.262028 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:43.517964 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:43.919742 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:44.020141 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:44.281092 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:44.288811 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:44.800092 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:44.955181 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:45.067481 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:45.323365 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240430 20:12:45.327572 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:45.589048 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:45.783133 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240430 20:12:45.855334 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:45.984592 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:46.385641 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:46.386848 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:46.420295 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:46.927261 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:47.024456 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:47.070685 29117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 20:12:47.456291 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:47.471810 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:47.485162 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 20:12:47.745371 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:48.054315 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:48.171496 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240430 20:12:48.271728 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:48.544461 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:48.568570 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:49.094323 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:49.098415 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:49.401844 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:49.647630 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:49.926126 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:49.950752 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240430 20:12:50.133705 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:50.241986 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:50.698237 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 20:12:50.772009 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:50.803241 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:51.094012 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:51.167585 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:51.627235 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:51.912052 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:51.952378 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:52.167105 29117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 20:12:52.167974 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240430 20:12:52.226990 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:52.488698 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:52.816922 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:53.072062 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:53.268922 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240430 20:12:53.280668 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:53.361456 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:53.692723 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:54.239990 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:54.258435 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:54.329102 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:54.579788 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:54.597849 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 20:12:55.127043 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:55.212536 29733 raft_consensus.cc:482] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:55.212821 29733 raft_consensus.cc:504] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:55.213825 29733 leader_election.cc:288] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:55.215581 29117 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:55.215736 29118 leader_election.cc:334] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:55.215919 29118 leader_election.cc:302] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:55.216279 29733 raft_consensus.cc:2726] T b8b2e9cef5024ea8b1f1d5a145331641 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:12:55.375698 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:55.425776 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:55.472048 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:55.689145 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240430 20:12:56.025967 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:56.378597 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:56.413219 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:56.628743 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:56.928903 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:57.037068 29740 raft_consensus.cc:482] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 142855f2169447a29ccb6ba6a9a7fe46)
I20240430 20:12:57.037246 29740 raft_consensus.cc:504] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999f86490d934d6c9f2f438b6000f584" member_type: VOTER last_known_addr { host: "127.28.82.1" port: 43573 } } peers { permanent_uuid: "142855f2169447a29ccb6ba6a9a7fe46" member_type: VOTER last_known_addr { host: "127.28.82.2" port: 36545 } } peers { permanent_uuid: "b3858d6e040645898f743997947970ad" member_type: VOTER last_known_addr { host: "127.28.82.3" port: 46029 } }
I20240430 20:12:57.038010 29740 leader_election.cc:288] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545), b3858d6e040645898f743997947970ad (127.28.82.3:46029)
W20240430 20:12:57.039849 29117 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:12:57.039994 29118 leader_election.cc:334] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:12:57.040076 29118 leader_election.cc:302] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 999f86490d934d6c9f2f438b6000f584; no voters: 142855f2169447a29ccb6ba6a9a7fe46, b3858d6e040645898f743997947970ad
I20240430 20:12:57.040362 29740 raft_consensus.cc:2726] T 18b59309bf554ce5a8d95e3a923f1b97 P 999f86490d934d6c9f2f438b6000f584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:12:57.227407 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240430 20:12:57.230180 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240430 20:12:57.289822 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:57.453516 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:57.842799 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:57.865167 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:58.210448 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240430 20:12:58.212806 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:58.484673 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:58.766788 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:12:59.075927 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:12:59.141337 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:12:59.530179 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:12:59.701556 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:00.077944 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:00.176175 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240430 20:13:00.328867 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:13:00.559278 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:00.637933 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240430 20:13:00.645325 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:01.025321 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:01.343161 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:13:01.597376 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:01.602361 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:01.982774 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:02.295748 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 20:13:02.553426 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:02.570741 29494 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
I20240430 20:13:02.648569 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:02.760000 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240430 20:13:02.953914 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:03.439960 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240430 20:13:03.520256 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:03.606599 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
I20240430 20:13:03.679342 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:03.932046 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:04.499047 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:04.721294 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:04.824873 29494 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica b3858d6e040645898f743997947970ad (127.28.82.3:46029) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111)
W20240430 20:13:04.918463 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:05.239038 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240430 20:13:05.484259 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:05.761858 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:05.836319 29493 meta_cache.cc:302] tablet b8b2e9cef5024ea8b1f1d5a145331641: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:13:05.911559 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:06.274166 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240430 20:13:06.477944 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:06.777763 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:06.912937 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:07.151093 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:13:07.420243 29118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240430 20:13:07.487253 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:07.825734 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:07.926685 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:07.944581 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240430 20:13:08.163494 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:13:08.499550 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:08.835088 29118 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer b3858d6e040645898f743997947970ad (127.28.82.3:46029): Couldn't send request to peer b3858d6e040645898f743997947970ad. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.3:46029: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240430 20:13:08.884590 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:08.942795 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:09.183784 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:13:09.526373 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
I20240430 20:13:09.935878 29536 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:13:09.969733 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:10.215785 29493 meta_cache.cc:302] tablet 18b59309bf554ce5a8d95e3a923f1b97: replica 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545) has failed: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111)
W20240430 20:13:10.510275 29117 consensus_peers.cc:485] T b8b94089cd004f6aab04ea4d067bdb82 P 999f86490d934d6c9f2f438b6000f584 -> Peer 142855f2169447a29ccb6ba6a9a7fe46 (127.28.82.2:36545): Couldn't send request to peer 142855f2169447a29ccb6ba6a9a7fe46. Status: Network error: Client connection negotiation failed: client connection to 127.28.82.2:36545: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240430 20:13:10.557920 29144 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50918: Illegal state: replica 999f86490d934d6c9f2f438b6000f584 is not leader of this config: current role FOLLOWER
W20240430 20:13:10.656922 29494 batcher.cc:441] Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.657732 29000 client-test-util.cc:49] 8 failed ops. First 10 errors follow
I20240430 20:13:10.657886 29000 client-test-util.cc:55] Op INSERT int64 rand_key=4747694291946176518, int64 link_to=1612007383946166278, int64 insert_ts=1714507930379440 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658104 29000 client-test-util.cc:55] Op INSERT int64 rand_key=3320104595893256202, int64 link_to=-3737690626343174134, int64 insert_ts=1714507930379661 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658210 29000 client-test-util.cc:55] Op INSERT int64 rand_key=5694092479354699789, int64 link_to=-2931475828137590771, int64 insert_ts=1714507930379819 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658278 29000 client-test-util.cc:55] Op INSERT int64 rand_key=4266362045813489681, int64 link_to=-8281173838426931183, int64 insert_ts=1714507930380070 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658366 29000 client-test-util.cc:55] Op INSERT int64 rand_key=5212760233222012952, int64 link_to=5621946285710573592, int64 insert_ts=1714507930380293 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658437 29000 client-test-util.cc:55] Op INSERT int64 rand_key=3784748324704092188, int64 link_to=272248275421233180, int64 insert_ts=1714507930380416 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658526 29000 client-test-util.cc:55] Op INSERT int64 rand_key=4731146512112615459, int64 link_to=-4271375674150813661, int64 insert_ts=1714507930380664 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
I20240430 20:13:10.658605 29000 client-test-util.cc:55] Op INSERT int64 rand_key=3303416078571405351, int64 link_to=8825951864246108199, int64 insert_ts=1714507930380817 had status Timed out: Failed to write batch of 8 ops to tablet 18b59309bf554ce5a8d95e3a923f1b97 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 18b59309bf554ce5a8d95e3a923f1b97, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.28.82.2:36545 (error 108)
F20240430 20:13:10.658692 29000 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714507990 (unix time) try "date -d @1714507990" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007148) received by PID 29000 (TID 0x7f13d3097d80) from PID 29000; stack trace: ***
@ 0x7f13daaa4980 (unknown) at ??:0
@ 0x7f13d3b4ffb7 gsignal at ??:0
@ 0x7f13d3b51921 abort at ??:0
@ 0x7f13d6878dcd google::LogMessage::Fail() at ??:0
@ 0x7f13d687cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f13d68787cc google::LogMessage::Flush() at ??:0
@ 0x7f13d6879f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f13e630eefd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed8c2 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test-util.h:?
@ 0x3d6588 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8889 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7f13d87130ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f13d8707bdb testing::Test::Run() at ??:0
@ 0x7f13d8707d9d testing::TestInfo::Run() at ??:0
@ 0x7f13d8708377 testing::TestSuite::Run() at ??:0
@ 0x7f13d870877c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f13d871360d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f13d8707e63 testing::UnitTest::Run() at ??:0
@ 0x7f13ed3f23c2 RUN_ALL_TESTS() at ??:0
@ 0x7f13ed3f08d9 main at ??:0
@ 0x7f13d3b32bf7 __libc_start_main at ??:0
@ 0x2ebf5a _start at ??:?