[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:34.575131 14564 test_util.cc:268] Using random seed: 950665969
I20240502 08:16:34.586274 14564 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 08:16:34.586357 14564 linked_list-test.cc:101] --------------
I20240502 08:16:34.586403 14564 linked_list-test.cc:102] 50 chains
I20240502 08:16:34.586463 14564 linked_list-test.cc:103] 3 tablets
I20240502 08:16:34.586524 14564 linked_list-test.cc:104] Mutations on
I20240502 08:16:34.586575 14564 linked_list-test.cc:105] --------------
I20240502 08:16:34.586792 14564 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:34.586848 14564 ts_itest-base.cc:113] --------------
I20240502 08:16:34.586913 14564 ts_itest-base.cc:114] 3 tablet servers
I20240502 08:16:34.586961 14564 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:34.587019 14564 ts_itest-base.cc:116] --------------
2024-05-02T08:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:34Z Disabled control of system clock
I20240502 08:16:34.608223 14564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
/tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.62:36635
--webserver_interface=127.14.57.62
--webserver_port=0
--builtin_ntp_servers=127.14.57.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.57.62:36635
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 08:16:34.923127 14577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:34.923425 14577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:34.923559 14577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:34.931192 14577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:34.931303 14577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:34.931375 14577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:34.931442 14577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:34.940176 14577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.57.62:36635
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.62:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.14.57.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:34.941517 14577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:34.943228 14577 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:34.954073 14583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.954600 14585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.954999 14582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:34.955550 14577 server_base.cc:1034] running on GCE node
I20240502 08:16:34.956271 14577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:34.957675 14577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:34.958889 14577 hybrid_clock.cc:648] HybridClock initialized: now 1714637794958817 us; error 49 us; skew 500 ppm
I20240502 08:16:34.959197 14577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:34.967228 14577 webserver.cc:460] Webserver started at http://127.14.57.62:44359/ using document root <none> and password file <none>
I20240502 08:16:34.967939 14577 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:34.968070 14577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:34.968380 14577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:34.971081 14577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data/instance:
uuid: "5666b5d810b7486080109a6dffdc2fc6"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-k8r3"
I20240502 08:16:34.971794 14577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal/instance:
uuid: "5666b5d810b7486080109a6dffdc2fc6"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-k8r3"
I20240502 08:16:34.976747 14577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240502 08:16:34.980435 14591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.981698 14577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:34.981843 14577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal
uuid: "5666b5d810b7486080109a6dffdc2fc6"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-k8r3"
I20240502 08:16:34.982008 14577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.006701 14577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.007534 14577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.007802 14577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.025146 14577 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.62:36635
I20240502 08:16:35.025162 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.62:36635 every 8 connection(s)
I20240502 08:16:35.026520 14577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/data/info.pb
I20240502 08:16:35.030149 14564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu as pid 14577
I20240502 08:16:35.030439 14564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/master-0/wal/instance
I20240502 08:16:35.030551 14645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.041306 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6: Bootstrap starting.
I20240502 08:16:35.044100 14645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.045028 14645 log.cc:824] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.047598 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6: No bootstrap required, opened a new log
I20240502 08:16:35.054073 14645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } }
I20240502 08:16:35.054394 14645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.054497 14645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5666b5d810b7486080109a6dffdc2fc6, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.055089 14645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } }
I20240502 08:16:35.055284 14645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:35.055404 14645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:35.055662 14645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.060150 14645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } }
I20240502 08:16:35.060607 14645 leader_election.cc:302] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5666b5d810b7486080109a6dffdc2fc6; no voters:
I20240502 08:16:35.061126 14645 leader_election.cc:288] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:35.061273 14650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.062410 14650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 5666b5d810b7486080109a6dffdc2fc6, State: Running, Role: LEADER
I20240502 08:16:35.062935 14650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } }
I20240502 08:16:35.063817 14645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:35.065801 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5666b5d810b7486080109a6dffdc2fc6. Latest consensus state: current_term: 1 leader_uuid: "5666b5d810b7486080109a6dffdc2fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } } }
I20240502 08:16:35.065829 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5666b5d810b7486080109a6dffdc2fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5666b5d810b7486080109a6dffdc2fc6" member_type: VOTER last_known_addr { host: "127.14.57.62" port: 36635 } } }
I20240502 08:16:35.066092 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.066131 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.067608 14655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:35.073925 14655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:35.084108 14655 catalog_manager.cc:1347] Generated new cluster ID: 27129b50c1894aefbaf3473210f8f8a0
I20240502 08:16:35.084246 14655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:35.100682 14655 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:35.101963 14655 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:35.109423 14655 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5666b5d810b7486080109a6dffdc2fc6: Generated new TSK 0
I20240502 08:16:35.110271 14655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:35.119751 14564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
/tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.1:0
--local_ip_for_outbound_sockets=127.14.57.1
--webserver_interface=127.14.57.1
--webserver_port=0
--tserver_master_addrs=127.14.57.62:36635
--builtin_ntp_servers=127.14.57.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 08:16:35.515726 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:35.516021 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:35.516186 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:35.523627 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:35.523777 14669 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 08:16:35.523838 14669 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 08:16:35.523927 14669 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 08:16:35.524037 14669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.1
I20240502 08:16:35.532902 14669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.14.57.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.14.57.62:36635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.1
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:35.534176 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.535861 14669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.548660 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.549053 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.549489 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.550031 14669 server_base.cc:1034] running on GCE node
I20240502 08:16:35.550698 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:35.551674 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:35.552906 14669 hybrid_clock.cc:648] HybridClock initialized: now 1714637795552825 us; error 46 us; skew 500 ppm
I20240502 08:16:35.553308 14669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.555919 14669 webserver.cc:460] Webserver started at http://127.14.57.1:35947/ using document root <none> and password file <none>
I20240502 08:16:35.556993 14669 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.557178 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.557554 14669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.560518 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data/instance:
uuid: "2fc6868d97f74b6c932851433570b3ba"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-k8r3"
I20240502 08:16:35.561316 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal/instance:
uuid: "2fc6868d97f74b6c932851433570b3ba"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-k8r3"
I20240502 08:16:35.566936 14669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:35.570897 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.572252 14669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:35.572434 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal
uuid: "2fc6868d97f74b6c932851433570b3ba"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-k8r3"
I20240502 08:16:35.572672 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.592343 14669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.593492 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.593889 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.595038 14669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.597085 14669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.597255 14669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.597395 14669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.597512 14669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.643158 14669 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.1:44973
I20240502 08:16:35.643200 14796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.1:44973 every 8 connection(s)
I20240502 08:16:35.644431 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/data/info.pb
I20240502 08:16:35.648434 14564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu as pid 14669
I20240502 08:16:35.648730 14564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-0/wal/instance
I20240502 08:16:35.652561 14564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
/tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.2:0
--local_ip_for_outbound_sockets=127.14.57.2
--webserver_interface=127.14.57.2
--webserver_port=0
--tserver_master_addrs=127.14.57.62:36635
--builtin_ntp_servers=127.14.57.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 08:16:35.661738 14797 heartbeater.cc:347] Connected to a master server at 127.14.57.62:36635
I20240502 08:16:35.662076 14797 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.663161 14797 heartbeater.cc:511] Master 127.14.57.62:36635 requested a full tablet report, sending...
I20240502 08:16:35.665773 14609 ts_manager.cc:194] Registered new tserver with Master: 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973)
I20240502 08:16:35.667889 14609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.1:57125
W20240502 08:16:35.979092 14801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:35.979377 14801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:35.979504 14801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:35.987315 14801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:35.987449 14801 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 08:16:35.987493 14801 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 08:16:35.987550 14801 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 08:16:35.987624 14801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.2
I20240502 08:16:35.996711 14801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.14.57.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.14.57.62:36635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.2
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:35.997957 14801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.999552 14801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.010988 14807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.011323 14806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.011710 14809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.012295 14801 server_base.cc:1034] running on GCE node
I20240502 08:16:36.012809 14801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.013690 14801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.014883 14801 hybrid_clock.cc:648] HybridClock initialized: now 1714637796014837 us; error 36 us; skew 500 ppm
I20240502 08:16:36.015174 14801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.017135 14801 webserver.cc:460] Webserver started at http://127.14.57.2:42421/ using document root <none> and password file <none>
I20240502 08:16:36.017822 14801 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.017923 14801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.018213 14801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.020633 14801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data/instance:
uuid: "5497377f4ed44a4284f486249e37c22d"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.021390 14801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal/instance:
uuid: "5497377f4ed44a4284f486249e37c22d"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.026129 14801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:36.029265 14815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.030586 14801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:36.030728 14801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal
uuid: "5497377f4ed44a4284f486249e37c22d"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.030898 14801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.046525 14801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.047282 14801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.047542 14801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.048435 14801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.050292 14801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.050385 14801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.050485 14801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.050554 14801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.081424 14801 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.2:39821
I20240502 08:16:36.081441 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.2:39821 every 8 connection(s)
I20240502 08:16:36.082672 14801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/data/info.pb
I20240502 08:16:36.088757 14564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu as pid 14801
I20240502 08:16:36.088919 14564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-1/wal/instance
I20240502 08:16:36.091709 14564 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
/tmp/dist-test-taskpxS8CY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.3:0
--local_ip_for_outbound_sockets=127.14.57.3
--webserver_interface=127.14.57.3
--webserver_port=0
--tserver_master_addrs=127.14.57.62:36635
--builtin_ntp_servers=127.14.57.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 08:16:36.094899 14931 heartbeater.cc:347] Connected to a master server at 127.14.57.62:36635
I20240502 08:16:36.095155 14931 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.095861 14931 heartbeater.cc:511] Master 127.14.57.62:36635 requested a full tablet report, sending...
I20240502 08:16:36.097105 14609 ts_manager.cc:194] Registered new tserver with Master: 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821)
I20240502 08:16:36.097827 14609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.2:47527
W20240502 08:16:36.406257 14934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.406517 14934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.406666 14934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.413769 14934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.413915 14934 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 08:16:36.413972 14934 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 08:16:36.414064 14934 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 08:16:36.414165 14934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.3
I20240502 08:16:36.423365 14934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.14.57.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.14.57.62:36635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.3
--log_dir=/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.424643 14934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.426379 14934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.438467 14939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.439052 14942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.439499 14940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.439975 14934 server_base.cc:1034] running on GCE node
I20240502 08:16:36.440549 14934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.441527 14934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.442757 14934 hybrid_clock.cc:648] HybridClock initialized: now 1714637796442707 us; error 36 us; skew 500 ppm
I20240502 08:16:36.443073 14934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.445315 14934 webserver.cc:460] Webserver started at http://127.14.57.3:43431/ using document root <none> and password file <none>
I20240502 08:16:36.446059 14934 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.446159 14934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.446468 14934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.449031 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data/instance:
uuid: "be64d0fa0ce74ef082a843b34d73d96b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.449781 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal/instance:
uuid: "be64d0fa0ce74ef082a843b34d73d96b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.454819 14934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:36.458083 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.459492 14934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 08:16:36.459627 14934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal
uuid: "be64d0fa0ce74ef082a843b34d73d96b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-k8r3"
I20240502 08:16:36.459791 14934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.487219 14934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.488054 14934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.488312 14934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.489698 14934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.491077 14934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.491165 14934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.491240 14934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.491289 14934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.522778 14934 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.3:44131
I20240502 08:16:36.522819 15061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.3:44131 every 8 connection(s)
I20240502 08:16:36.524113 14934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/data/info.pb
I20240502 08:16:36.525254 14564 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpxS8CY/build/asan/bin/kudu as pid 14934
I20240502 08:16:36.525444 14564 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpxS8CY/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714637794503062-14564-0/linked-list-cluster/ts-2/wal/instance
I20240502 08:16:36.537464 15062 heartbeater.cc:347] Connected to a master server at 127.14.57.62:36635
I20240502 08:16:36.537727 15062 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.538471 15062 heartbeater.cc:511] Master 127.14.57.62:36635 requested a full tablet report, sending...
I20240502 08:16:36.539702 14609 ts_manager.cc:194] Registered new tserver with Master: be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:36.540479 14609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.3:41857
I20240502 08:16:36.541486 14564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 08:16:36.561843 14609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43772:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 08:16:36.567715 14609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.606557 14863 tablet_service.cc:1467] Processing CreateTablet for tablet 301b0b861ed745d4b5c46cad4d7673d9 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 08:16:36.607671 14862 tablet_service.cc:1467] Processing CreateTablet for tablet ae01fc8e080f4404831074653d1f293c (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 08:16:36.608503 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae01fc8e080f4404831074653d1f293c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.607321 14864 tablet_service.cc:1467] Processing CreateTablet for tablet 04285e042f0646a99490ef090adbb766 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 08:16:36.606384 14996 tablet_service.cc:1467] Processing CreateTablet for tablet 04285e042f0646a99490ef090adbb766 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 08:16:36.609377 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04285e042f0646a99490ef090adbb766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.606366 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 301b0b861ed745d4b5c46cad4d7673d9 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 08:16:36.611443 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04285e042f0646a99490ef090adbb766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.610801 14730 tablet_service.cc:1467] Processing CreateTablet for tablet ae01fc8e080f4404831074653d1f293c (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 08:16:36.612470 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae01fc8e080f4404831074653d1f293c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.610729 14731 tablet_service.cc:1467] Processing CreateTablet for tablet 301b0b861ed745d4b5c46cad4d7673d9 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 08:16:36.610769 14732 tablet_service.cc:1467] Processing CreateTablet for tablet 04285e042f0646a99490ef090adbb766 (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 08:16:36.613747 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301b0b861ed745d4b5c46cad4d7673d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.614182 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301b0b861ed745d4b5c46cad4d7673d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.615345 14732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04285e042f0646a99490ef090adbb766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.624297 15080 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Bootstrap starting.
I20240502 08:16:36.626809 15081 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Bootstrap starting.
I20240502 08:16:36.627151 14863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301b0b861ed745d4b5c46cad4d7673d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.627820 15080 tablet_bootstrap.cc:654] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.629133 15080 log.cc:824] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:36.630864 15081 tablet_bootstrap.cc:654] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.632786 15081 log.cc:824] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:36.635836 14994 tablet_service.cc:1467] Processing CreateTablet for tablet ae01fc8e080f4404831074653d1f293c (DEFAULT_TABLE table=TestTable [id=265a87da5749474686a3090c12671a62]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 08:16:36.636540 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae01fc8e080f4404831074653d1f293c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.637115 15081 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: No bootstrap required, opened a new log
I20240502 08:16:36.637540 15081 ts_tablet_manager.cc:1367] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:16:36.637953 15080 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: No bootstrap required, opened a new log
I20240502 08:16:36.638257 15080 ts_tablet_manager.cc:1367] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Time spent bootstrapping tablet: real 0.014s user 0.014s sys 0.000s
I20240502 08:16:36.648671 15080 raft_consensus.cc:348] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.649327 15080 raft_consensus.cc:374] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.649484 15080 raft_consensus.cc:729] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5497377f4ed44a4284f486249e37c22d, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.650328 15080 consensus_queue.cc:260] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.651501 15084 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Bootstrap starting.
I20240502 08:16:36.651723 14931 heartbeater.cc:503] Master 127.14.57.62:36635 was elected leader, sending a full tablet report...
I20240502 08:16:36.652343 15080 ts_tablet_manager.cc:1398] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20240502 08:16:36.652799 15080 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: Bootstrap starting.
I20240502 08:16:36.654400 15080 tablet_bootstrap.cc:654] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.655436 15084 tablet_bootstrap.cc:654] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.656785 15084 log.cc:824] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:36.658658 15081 raft_consensus.cc:348] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.660383 15081 raft_consensus.cc:374] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.660552 15081 raft_consensus.cc:729] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc6868d97f74b6c932851433570b3ba, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.661412 15081 consensus_queue.cc:260] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.662328 15084 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: No bootstrap required, opened a new log
I20240502 08:16:36.662679 15084 ts_tablet_manager.cc:1367] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.005s
I20240502 08:16:36.665776 14797 heartbeater.cc:503] Master 127.14.57.62:36635 was elected leader, sending a full tablet report...
I20240502 08:16:36.666342 15081 ts_tablet_manager.cc:1398] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Time spent starting tablet: real 0.029s user 0.011s sys 0.005s
I20240502 08:16:36.666671 15080 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: No bootstrap required, opened a new log
I20240502 08:16:36.666816 15081 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: Bootstrap starting.
I20240502 08:16:36.667376 15080 ts_tablet_manager.cc:1367] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240502 08:16:36.668304 15080 raft_consensus.cc:348] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.668622 15080 raft_consensus.cc:374] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.668723 15081 tablet_bootstrap.cc:654] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.668746 15080 raft_consensus.cc:729] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5497377f4ed44a4284f486249e37c22d, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.669026 15080 consensus_queue.cc:260] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.669713 15080 ts_tablet_manager.cc:1398] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:36.670018 15080 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d: Bootstrap starting.
I20240502 08:16:36.671684 15080 tablet_bootstrap.cc:654] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.672235 15084 raft_consensus.cc:348] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.672827 15084 raft_consensus.cc:374] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.672986 15084 raft_consensus.cc:729] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.673972 15084 consensus_queue.cc:260] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.676812 15084 ts_tablet_manager.cc:1398] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20240502 08:16:36.677243 15081 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: No bootstrap required, opened a new log
I20240502 08:16:36.677366 15062 heartbeater.cc:503] Master 127.14.57.62:36635 was elected leader, sending a full tablet report...
I20240502 08:16:36.677398 15081 ts_tablet_manager.cc:1367] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240502 08:16:36.678187 15080 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d: No bootstrap required, opened a new log
I20240502 08:16:36.678388 15080 ts_tablet_manager.cc:1367] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20240502 08:16:36.678272 15081 raft_consensus.cc:348] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.678426 15084 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b: Bootstrap starting.
I20240502 08:16:36.678471 15081 raft_consensus.cc:374] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.678555 15081 raft_consensus.cc:729] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc6868d97f74b6c932851433570b3ba, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.678855 15081 consensus_queue.cc:260] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.679095 15080 raft_consensus.cc:348] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.679332 15080 raft_consensus.cc:374] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.679415 15080 raft_consensus.cc:729] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5497377f4ed44a4284f486249e37c22d, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.679725 15080 consensus_queue.cc:260] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.680267 15080 ts_tablet_manager.cc:1398] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.680531 15081 ts_tablet_manager.cc:1398] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:36.680579 15084 tablet_bootstrap.cc:654] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.680826 15081 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Bootstrap starting.
I20240502 08:16:36.682512 15081 tablet_bootstrap.cc:654] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.684643 15081 tablet_bootstrap.cc:492] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: No bootstrap required, opened a new log
I20240502 08:16:36.684789 15081 ts_tablet_manager.cc:1367] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:36.685588 15084 tablet_bootstrap.cc:492] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b: No bootstrap required, opened a new log
I20240502 08:16:36.685779 15084 ts_tablet_manager.cc:1367] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:16:36.686056 15081 raft_consensus.cc:348] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.686350 15081 raft_consensus.cc:374] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.686436 15081 raft_consensus.cc:729] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc6868d97f74b6c932851433570b3ba, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.686582 15084 raft_consensus.cc:348] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.686741 15084 raft_consensus.cc:374] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.686686 15081 consensus_queue.cc:260] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.686829 15084 raft_consensus.cc:729] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.687084 15084 consensus_queue.cc:260] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.687199 15081 ts_tablet_manager.cc:1398] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:36.687587 15084 ts_tablet_manager.cc:1398] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.687851 15084 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b: Bootstrap starting.
I20240502 08:16:36.689427 15084 tablet_bootstrap.cc:654] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.691236 15084 tablet_bootstrap.cc:492] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b: No bootstrap required, opened a new log
I20240502 08:16:36.691363 15084 ts_tablet_manager.cc:1367] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:36.691848 15084 raft_consensus.cc:348] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.692008 15084 raft_consensus.cc:374] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.692076 15084 raft_consensus.cc:729] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.692284 15084 consensus_queue.cc:260] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.692673 15084 ts_tablet_manager.cc:1398] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 08:16:36.775919 15063 tablet.cc:2369] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:36.834596 14932 tablet.cc:2369] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:36.891157 15091 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.891417 15091 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.893118 15091 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821)
I20240502 08:16:36.896601 15085 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.896878 15085 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
W20240502 08:16:36.897243 14798 tablet.cc:2369] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:36.898955 15085 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:36.900233 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:36.901335 14752 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:36.901997 14950 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:36.902081 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:36.902387 15091 raft_consensus.cc:2781] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:36.902521 15091 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:36.902608 15091 raft_consensus.cc:3037] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.902618 14884 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:36.905395 15091 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.906059 15091 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Requested vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821)
I20240502 08:16:36.906481 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d"
I20240502 08:16:36.906656 14884 raft_consensus.cc:3037] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.907860 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba"
I20240502 08:16:36.908048 14752 raft_consensus.cc:3037] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.908404 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:36.909097 14884 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:36.909443 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be64d0fa0ce74ef082a843b34d73d96b" is_pre_election: true
I20240502 08:16:36.909591 14950 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:36.909899 15017 raft_consensus.cc:2379] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5497377f4ed44a4284f486249e37c22d in current term 1: Already voted for candidate be64d0fa0ce74ef082a843b34d73d96b in this term.
I20240502 08:16:36.910172 15091 raft_consensus.cc:2781] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:36.910631 14818 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d; no voters: 2fc6868d97f74b6c932851433570b3ba, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:36.910935 14752 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:36.911020 15085 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.911397 15091 raft_consensus.cc:686] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [term 1 LEADER]: Becoming Leader. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Running, Role: LEADER
I20240502 08:16:36.911855 15091 consensus_queue.cc:237] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:36.916445 14608 catalog_manager.cc:5484] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b reported cstate change: term changed from 0 to 1, leader changed from <none> to be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3). New cstate: current_term: 1 leader_uuid: "be64d0fa0ce74ef082a843b34d73d96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:36.960811 15091 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.960997 15091 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.961567 15091 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973)
I20240502 08:16:36.962075 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:36.962142 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:36.962298 14884 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:36.962405 14752 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:36.962679 14950 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:36.962931 15091 raft_consensus.cc:2781] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:36.963016 15091 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:36.963125 15091 raft_consensus.cc:3037] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.966941 15091 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.967446 15091 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Requested vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973)
I20240502 08:16:36.967850 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d"
I20240502 08:16:36.967868 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba"
I20240502 08:16:36.968041 14884 raft_consensus.cc:3037] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.968042 14752 raft_consensus.cc:3037] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.973160 14884 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:36.973176 14752 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:36.973562 14950 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:36.973834 15091 raft_consensus.cc:2781] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:36.973990 15091 raft_consensus.cc:686] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [term 1 LEADER]: Becoming Leader. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Running, Role: LEADER
I20240502 08:16:36.974222 15091 consensus_queue.cc:237] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:36.976553 14608 catalog_manager.cc:5484] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b reported cstate change: term changed from 0 to 1, leader changed from <none> to be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3). New cstate: current_term: 1 leader_uuid: "be64d0fa0ce74ef082a843b34d73d96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:37.067430 15091 raft_consensus.cc:482] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.067641 15091 raft_consensus.cc:504] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:37.068289 15091 leader_election.cc:288] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821)
I20240502 08:16:37.068897 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:37.068912 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:37.069114 14884 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:37.069129 14752 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 0.
I20240502 08:16:37.069540 14950 leader_election.cc:302] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:37.069777 15091 raft_consensus.cc:2781] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.069939 15091 raft_consensus.cc:482] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.070026 15091 raft_consensus.cc:3037] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.071811 15091 raft_consensus.cc:504] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:37.072321 15091 leader_election.cc:288] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Requested vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821)
I20240502 08:16:37.072836 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba"
I20240502 08:16:37.072861 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "be64d0fa0ce74ef082a843b34d73d96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d"
I20240502 08:16:37.073009 14752 raft_consensus.cc:3037] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.073014 14884 raft_consensus.cc:3037] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.074790 14884 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:37.074831 14752 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64d0fa0ce74ef082a843b34d73d96b in term 1.
I20240502 08:16:37.075165 14950 leader_election.cc:302] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b; no voters:
I20240502 08:16:37.075423 15091 raft_consensus.cc:2781] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.075562 15091 raft_consensus.cc:686] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [term 1 LEADER]: Becoming Leader. State: Replica: be64d0fa0ce74ef082a843b34d73d96b, State: Running, Role: LEADER
I20240502 08:16:37.075827 15091 consensus_queue.cc:237] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:37.078379 14608 catalog_manager.cc:5484] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b reported cstate change: term changed from 0 to 1, leader changed from <none> to be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3). New cstate: current_term: 1 leader_uuid: "be64d0fa0ce74ef082a843b34d73d96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.142063 14564 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 08:16:37.144680 14564 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 2fc6868d97f74b6c932851433570b3ba to finish bootstrapping
I20240502 08:16:37.153510 14564 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5497377f4ed44a4284f486249e37c22d to finish bootstrapping
I20240502 08:16:37.160437 14564 ts_itest-base.cc:233] Waiting for 3 tablets on tserver be64d0fa0ce74ef082a843b34d73d96b to finish bootstrapping
I20240502 08:16:37.168747 15103 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.14.57.2:42421/threadz
http://127.14.57.2:42421/maintenance-manager
http://127.14.57.1:35947/stacks
http://127.14.57.3:43431/mem-trackers
http://127.14.57.2:42421/metrics
http://127.14.57.62:44359/mem-trackers
http://127.14.57.3:43431/metrics
http://127.14.57.3:43431/startup
http://127.14.57.62:44359/dump-entities
http://127.14.57.2:42421/tablets
http://127.14.57.62:44359/threadz
http://127.14.57.62:44359/ipki-ca-cert
http://127.14.57.2:42421/startup
http://127.14.57.1:35947/scans
http://127.14.57.2:42421/threadz?group=all
http://127.14.57.62:44359/tablet-servers
http://127.14.57.3:43431/transactions?tablet_id=ae01fc8e080f4404831074653d1f293c
http://127.14.57.1:35947/mem-trackers
http://127.14.57.62:44359/startup
http://127.14.57.62:44359/metrics
http://127.14.57.62:44359/masters
http://127.14.57.1:35947/startup
http://127.14.57.1:35947/tablets
http://127.14.57.1:35947/metrics
http://127.14.57.62:44359/threadz?group=all
http://127.14.57.62:44359/ipki-ca-cert-pem
http://127.14.57.2:42421/mem-trackers
http://127.14.57.3:43431/threadz
http://127.14.57.1:35947/threadz?group=all
http://127.14.57.3:43431/stacks
http://127.14.57.62:44359/ipki-ca-cert-der
http://127.14.57.3:43431/maintenance-manager
http://127.14.57.1:35947/maintenance-manager
http://127.14.57.2:42421/transactions?tablet_id=ae01fc8e080f4404831074653d1f293c
http://127.14.57.3:43431/tablets
http://127.14.57.1:35947/threadz
http://127.14.57.2:42421/scans
http://127.14.57.3:43431/scans
http://127.14.57.62:44359/stacks
http://127.14.57.2:42421/stacks
http://127.14.57.1:35947/transactions?tablet_id=ae01fc8e080f4404831074653d1f293c
http://127.14.57.3:43431/threadz?group=all
http://127.14.57.62:44359/tables
I20240502 08:16:37.173126 14564 linked_list-test-util.h:401] Running for: 30.000s
I20240502 08:16:37.173293 14564 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 08:16:37.173390 14564 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 08:16:37.218297 14752 raft_consensus.cc:1261] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.218333 14884 raft_consensus.cc:1261] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.219759 15098 consensus_queue.cc:1035] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.220553 15098 consensus_queue.cc:1035] T 301b0b861ed745d4b5c46cad4d7673d9 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.221102 14752 raft_consensus.cc:1261] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.221518 14751 raft_consensus.cc:1261] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.222199 14884 raft_consensus.cc:1261] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.222887 15091 consensus_queue.cc:1035] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.223345 15091 consensus_queue.cc:1035] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.225123 15112 consensus_queue.cc:1035] T ae01fc8e080f4404831074653d1f293c P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.226897 14882 raft_consensus.cc:1261] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Refusing update from remote peer be64d0fa0ce74ef082a843b34d73d96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.227443 15112 consensus_queue.cc:1035] T 04285e042f0646a99490ef090adbb766 P be64d0fa0ce74ef082a843b34d73d96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.290127 15109 mvcc.cc:205] Tried to move back new op lower bound from 7023156417402253312 to 7023156416824500224. Current Snapshot: MvccSnapshot[applied={T|T < 7023156417402253312}]
I20240502 08:16:37.945181 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 08:16:37.988427 15141 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fc57633edcd google::LogMessage::Fail() at ??:0
@ 0x7fc576342b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc57633e7cc google::LogMessage::Flush() at ??:0
@ 0x7fc57633ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc578f67c80 kudu::operator-() at ??:0
@ 0x7fc585be814a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fc585bb25de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fc585bb2336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fc585b5f2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fc58862ce47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fc58863b9fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fc58863b862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc58207102a std::function<>::operator()() at ??:0
@ 0x7fc5820697e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fc5820647f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fc582062927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fc5820a0089 handle_request at ??:0
@ 0x7fc5820a276b worker_thread at ??:0
@ 0x7fc57c3b36db start_thread at ??:0
@ 0x7fc5751f171f clone at ??:0
W20240502 08:16:38.210465 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: recv got EOF from 127.14.57.3:44131 (error 108)
W20240502 08:16:38.221433 14843 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.235213 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.248278 14843 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.258968 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.282891 14843 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.295886 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:38.299711 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:38.326764 14843 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.326872 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.330751 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.331943 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.332820 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.337328 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.340188 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.341539 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.348315 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.355872 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.376278 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.379519 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.380928 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.380990 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.388653 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.393364 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.393364 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.394635 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.418072 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.429894 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.431727 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.453887 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.463229 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.466876 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.469219 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.479828 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.501163 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.518366 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.524693 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.524981 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.530982 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.536939 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.573827 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.582063 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.592754 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.601678 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.604236 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.618919 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.656559 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.659729 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.666028 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.671360 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.684357 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.688163 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.740543 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.754120 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.755949 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.772159 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.780463 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.784113 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.790971 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.837786 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.863049 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.869400 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.877389 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.898830 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.936363 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:38.973819 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.975374 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:38.984522 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.014812 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.048849 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.086608 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.094069 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.094439 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.102447 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.137738 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.167109 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.171258 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.207115 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.212630 15067 meta_cache.cc:302] tablet ae01fc8e080f4404831074653d1f293c: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:39.226568 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.273437 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.290107 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.315842 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.333840 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.369603 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.418262 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.431005 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.441442 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:39.454077 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
W20240502 08:16:39.477226 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:39.478120 15169 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.478307 15169 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:39.479755 15169 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:39.481269 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.482178 15174 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.482372 15174 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
W20240502 08:16:39.483739 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.485184 15174 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131), 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973)
I20240502 08:16:39.485321 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:39.485563 14751 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5497377f4ed44a4284f486249e37c22d in term 1.
I20240502 08:16:39.486071 14817 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters:
I20240502 08:16:39.488122 15174 raft_consensus.cc:2781] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:39.488255 15174 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting leader election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.488351 15174 raft_consensus.cc:3037] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.492995 15174 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:39.493872 14882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "2fc6868d97f74b6c932851433570b3ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:39.493989 15174 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: Requested vote from peers be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131), 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973)
I20240502 08:16:39.494199 14882 raft_consensus.cc:2379] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fc6868d97f74b6c932851433570b3ba in current term 2: Already voted for candidate 5497377f4ed44a4284f486249e37c22d in this term.
I20240502 08:16:39.494549 15169 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.494699 15174 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.494794 15169 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.494868 15174 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.495438 15169 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:39.495791 15174 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:39.495963 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:39.496171 14750 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5497377f4ed44a4284f486249e37c22d in term 1.
I20240502 08:16:39.494580 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04285e042f0646a99490ef090adbb766" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba"
I20240502 08:16:39.496688 14751 raft_consensus.cc:3037] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.496650 14817 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters:
I20240502 08:16:39.496961 15174 raft_consensus.cc:2781] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:39.497079 15174 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting leader election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.497170 15174 raft_consensus.cc:3037] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.499851 14751 raft_consensus.cc:2454] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5497377f4ed44a4284f486249e37c22d in term 2.
I20240502 08:16:39.500164 15174 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.500439 14817 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters:
I20240502 08:16:39.500867 15174 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: Requested vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:39.501075 15174 raft_consensus.cc:2781] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:39.501246 15174 raft_consensus.cc:686] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [term 2 LEADER]: Becoming Leader. State: Replica: 5497377f4ed44a4284f486249e37c22d, State: Running, Role: LEADER
I20240502 08:16:39.501374 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba"
I20240502 08:16:39.501569 14750 raft_consensus.cc:3037] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.501756 15174 consensus_queue.cc:237] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:39.504447 14750 raft_consensus.cc:2454] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5497377f4ed44a4284f486249e37c22d in term 2.
I20240502 08:16:39.505043 14817 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters:
I20240502 08:16:39.505944 15169 raft_consensus.cc:482] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.506006 15176 raft_consensus.cc:2781] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:39.506150 15176 raft_consensus.cc:686] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [term 2 LEADER]: Becoming Leader. State: Replica: 5497377f4ed44a4284f486249e37c22d, State: Running, Role: LEADER
I20240502 08:16:39.506093 15169 raft_consensus.cc:504] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.506469 15176 consensus_queue.cc:237] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.506695 15169 leader_election.cc:288] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:39.507717 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:39.508688 14686 leader_election.cc:334] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.510969 15174 raft_consensus.cc:482] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
I20240502 08:16:39.511152 15174 raft_consensus.cc:504] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.511826 15174 leader_election.cc:288] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fc6868d97f74b6c932851433570b3ba (127.14.57.1:44973), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:39.513404 14818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.515362 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:39.515837 15169 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:39.518393 14882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301b0b861ed745d4b5c46cad4d7673d9" candidate_uuid: "2fc6868d97f74b6c932851433570b3ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:39.518415 14603 catalog_manager.cc:5484] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d reported cstate change: term changed from 1 to 2, leader changed from be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3) to 5497377f4ed44a4284f486249e37c22d (127.14.57.2). New cstate: current_term: 2 leader_uuid: "5497377f4ed44a4284f486249e37c22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.519229 14882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "2fc6868d97f74b6c932851433570b3ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d" is_pre_election: true
I20240502 08:16:39.519441 14882 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc6868d97f74b6c932851433570b3ba in term 1.
W20240502 08:16:39.519791 14818 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:39.520072 14818 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.520730 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:39.521421 15169 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:16:39.526114 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:39.526994 14685 leader_election.cc:302] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters: be64d0fa0ce74ef082a843b34d73d96b
W20240502 08:16:39.527367 14818 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.527632 15169 raft_consensus.cc:2781] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 08:16:39.527650 14818 leader_election.cc:334] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.527761 15169 raft_consensus.cc:482] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader be64d0fa0ce74ef082a843b34d73d96b)
W20240502 08:16:39.527772 14818 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.527853 15169 raft_consensus.cc:3037] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.531056 15169 raft_consensus.cc:504] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.532497 14882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "2fc6868d97f74b6c932851433570b3ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "5497377f4ed44a4284f486249e37c22d"
I20240502 08:16:39.532724 14882 raft_consensus.cc:3037] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:39.535395 14882 raft_consensus.cc:2454] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc6868d97f74b6c932851433570b3ba in term 2.
I20240502 08:16:39.535820 14685 leader_election.cc:302] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba, 5497377f4ed44a4284f486249e37c22d; no voters:
I20240502 08:16:39.536451 15185 raft_consensus.cc:2781] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:39.536408 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae01fc8e080f4404831074653d1f293c" candidate_uuid: "5497377f4ed44a4284f486249e37c22d" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2fc6868d97f74b6c932851433570b3ba" is_pre_election: true
I20240502 08:16:39.536952 15169 leader_election.cc:288] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 election: Requested vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
I20240502 08:16:39.537011 15185 raft_consensus.cc:686] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [term 2 LEADER]: Becoming Leader. State: Replica: 2fc6868d97f74b6c932851433570b3ba, State: Running, Role: LEADER
W20240502 08:16:39.538583 14686 leader_election.cc:334] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:39.538236 15185 consensus_queue.cc:237] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:39.541131 14817 leader_election.cc:302] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5497377f4ed44a4284f486249e37c22d; no voters: 2fc6868d97f74b6c932851433570b3ba, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:39.542902 15179 raft_consensus.cc:2726] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:39.543754 14604 catalog_manager.cc:5484] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba reported cstate change: term changed from 1 to 2, leader changed from be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3) to 2fc6868d97f74b6c932851433570b3ba (127.14.57.1). New cstate: current_term: 2 leader_uuid: "2fc6868d97f74b6c932851433570b3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.545965 14603 catalog_manager.cc:5484] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d reported cstate change: term changed from 1 to 2, leader changed from be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3) to 5497377f4ed44a4284f486249e37c22d (127.14.57.2). New cstate: current_term: 2 leader_uuid: "5497377f4ed44a4284f486249e37c22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.588016 14751 raft_consensus.cc:1261] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Refusing update from remote peer 5497377f4ed44a4284f486249e37c22d: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20240502 08:16:39.588748 15179 consensus_queue.cc:1035] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20240502 08:16:39.589069 14818 consensus_peers.cc:485] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:39.591146 14842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41444: Illegal state: replica 5497377f4ed44a4284f486249e37c22d is not leader of this config: current role FOLLOWER
I20240502 08:16:39.600520 15190 mvcc.cc:205] Tried to move back new op lower bound from 7023156427109441536 to 7023156426765615104. Current Snapshot: MvccSnapshot[applied={T|T < 7023156427103838208}]
I20240502 08:16:39.617275 14751 raft_consensus.cc:1261] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Refusing update from remote peer 5497377f4ed44a4284f486249e37c22d: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20240502 08:16:39.618247 15178 consensus_queue.cc:1035] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
W20240502 08:16:39.619043 14818 consensus_peers.cc:485] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:39.646289 14882 raft_consensus.cc:1261] T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d [term 2 FOLLOWER]: Refusing update from remote peer 2fc6868d97f74b6c932851433570b3ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20240502 08:16:39.646881 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:16:39.647020 15185 consensus_queue.cc:1035] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20240502 08:16:39.653337 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:39.658915 15198 mvcc.cc:205] Tried to move back new op lower bound from 7023156427109441536 to 7023156426765615104. Current Snapshot: MvccSnapshot[applied={T|T < 7023156427109441536}]
I20240502 08:16:39.832619 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:16:41.848204 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:16:42.018059 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:42.245754 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:16:42.414827 14818 consensus_peers.cc:485] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:16:42.449597 14818 consensus_peers.cc:485] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:16:42.847971 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling FlushMRSOp(04285e042f0646a99490ef090adbb766): perf score=1.000000
I20240502 08:16:42.918183 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling FlushMRSOp(04285e042f0646a99490ef090adbb766): perf score=1.000000
W20240502 08:16:43.521381 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.3:44131, user_credentials={real_user=slave}} blocked reactor thread for 44309us
W20240502 08:16:44.283015 14820 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.200s user 0.000s sys 0.000s
I20240502 08:16:44.305784 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: FlushMRSOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 1.456s user 0.526s sys 0.110s Metrics: {"bytes_written":147409,"cfile_init":1,"compiler_manager_pool.queue_time_us":536229,"compiler_manager_pool.run_cpu_time_us":649005,"compiler_manager_pool.run_wall_time_us":1343022,"dirs.queue_time_us":2638,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":86256,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3006,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8032,"thread_start_us":5167,"threads_started":2}
I20240502 08:16:44.307783 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766): 140945 bytes on disk
I20240502 08:16:44.308748 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 08:16:44.519317 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: FlushMRSOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 1.599s user 0.496s sys 0.169s Metrics: {"bytes_written":2319,"cfile_init":1,"compiler_manager_pool.queue_time_us":981,"delete_count":0,"dirs.queue_time_us":8641,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":57097,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2509,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8131,"thread_start_us":15994,"threads_started":2,"update_count":335}
I20240502 08:16:44.521183 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766): 141852 bytes on disk
I20240502 08:16:44.527396 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 08:16:44.528242 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling MajorDeltaCompactionOp(04285e042f0646a99490ef090adbb766): perf score=0.018028
I20240502 08:16:44.721550 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: MajorDeltaCompactionOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.193s user 0.048s sys 0.025s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3064,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":4112,"lbm_read_time_us":417,"lbm_reads_lt_1ms":20,"lbm_write_time_us":373,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":3859,"threads_started":2,"update_count":335}
W20240502 08:16:44.753230 14818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 08:16:44.754319 14818 consensus_peers.cc:485] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:16:44.923574 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:44.965317 14818 consensus_peers.cc:485] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:16:45.009016 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 08:16:45.010579 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:16:45.078234 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:45.139629 15188 log.cc:924] Time spent T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20240502 08:16:45.139699 15201 log.cc:924] Time spent T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20240502 08:16:45.139659 15192 log.cc:924] Time spent T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240502 08:16:45.140285 15200 log.cc:924] Time spent T ae01fc8e080f4404831074653d1f293c P 5497377f4ed44a4284f486249e37c22d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240502 08:16:47.183977 14564 linked_list-test-util.h:417] Sample at HT timestamp: 7023156458179592192 Inserted count: 22550
W20240502 08:16:47.263762 14818 consensus_peers.cc:485] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:16:47.285776 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:16:47.470273 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:47.537130 14818 consensus_peers.cc:485] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:16:47.567711 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling FlushMRSOp(04285e042f0646a99490ef090adbb766): perf score=1.000000
W20240502 08:16:47.618649 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:16:47.734598 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling FlushMRSOp(04285e042f0646a99490ef090adbb766): perf score=1.000000
I20240502 08:16:48.390522 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: FlushMRSOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.823s user 0.462s sys 0.118s Metrics: {"bytes_written":8141,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":14057,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":837,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":29383,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7892,"thread_start_us":13954,"threads_started":1,"update_count":1243}
I20240502 08:16:48.392328 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766): 130153 bytes on disk
I20240502 08:16:48.394488 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 08:16:48.395398 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling CompactRowSetsOp(04285e042f0646a99490ef090adbb766): perf score=1.014686
I20240502 08:16:48.518857 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: FlushMRSOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.784s user 0.472s sys 0.123s Metrics: {"bytes_written":9589,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6528,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":7467,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2397,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8024,"thread_start_us":6650,"threads_started":1,"update_count":1474}
I20240502 08:16:48.520566 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling CompactRowSetsOp(04285e042f0646a99490ef090adbb766): perf score=1.014817
I20240502 08:16:49.197607 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: CompactRowSetsOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.801s user 0.593s sys 0.025s Metrics: {"bytes_written":290989,"cfile_cache_hit":13,"cfile_cache_hit_bytes":94974,"cfile_cache_miss":59,"cfile_cache_miss_bytes":963918,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":6522,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":11097,"drs_written":1,"lbm_read_time_us":2780,"lbm_reads_lt_1ms":91,"lbm_write_time_us":3714,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":519014,"rows_written":15924,"spinlock_wait_cycles":896,"thread_start_us":6648,"threads_started":1}
I20240502 08:16:49.199553 14932 maintenance_manager.cc:419] P 5497377f4ed44a4284f486249e37c22d: Scheduling UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766): 293430 bytes on disk
I20240502 08:16:49.200178 14820 maintenance_manager.cc:643] P 5497377f4ed44a4284f486249e37c22d: UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 08:16:49.328238 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: CompactRowSetsOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.807s user 0.627s sys 0.019s Metrics: {"bytes_written":294513,"cfile_cache_hit":14,"cfile_cache_hit_bytes":96257,"cfile_cache_miss":62,"cfile_cache_miss_bytes":979567,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":17530,"dirs.run_cpu_time_us":838,"dirs.run_wall_time_us":4864,"drs_written":1,"lbm_read_time_us":9527,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":3275,"lbm_writes_lt_1ms":51,"mutex_wait_us":22,"num_input_rowsets":2,"peak_mem_usage":531007,"rows_written":16155,"spinlock_wait_cycles":17792,"thread_start_us":22448,"threads_started":5}
I20240502 08:16:49.329234 14798 maintenance_manager.cc:419] P 2fc6868d97f74b6c932851433570b3ba: Scheduling UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766): 298191 bytes on disk
I20240502 08:16:49.329854 14688 maintenance_manager.cc:643] P 2fc6868d97f74b6c932851433570b3ba: UndoDeltaBlockGCOp(04285e042f0646a99490ef090adbb766) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 08:16:49.776202 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:49.865598 14818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:16:49.868216 14818 consensus_peers.cc:485] T 04285e042f0646a99490ef090adbb766 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:16:49.922212 15104 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:50.083547 14818 consensus_peers.cc:485] T 301b0b861ed745d4b5c46cad4d7673d9 P 5497377f4ed44a4284f486249e37c22d -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:16:50.253669 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:16:50.254570 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:16:51.764093 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 08:16:51.904944 15104 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7efc9e477dcd google::LogMessage::Fail() at ??:0
@ 0x7efc9e47bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7efc9e4777cc google::LogMessage::Flush() at ??:0
@ 0x7efc9e478f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efca10a0c80 kudu::operator-() at ??:0
@ 0x7efcadd2114a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7efcadceb5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7efcadceb336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7efcadc982d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7efcb0765e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7efcb07749fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7efcb0774862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7efcaa1aa02a std::function<>::operator()() at ??:0
@ 0x7efcaa1a27e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7efcaa19d7f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7efcaa19b927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7efcaa1d9089 handle_request at ??:0
@ 0x7efcaa1db76b worker_thread at ??:0
@ 0x7efca44ec6db start_thread at ??:0
@ 0x7efc9d32a71f clone at ??:0
W20240502 08:16:52.144477 14685 connection.cc:535] client connection to 127.14.57.2:39821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:52.144706 15066 connection.cc:535] client connection to 127.14.57.2:39821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:52.145080 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:52.146206 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:52.148506 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.150432 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.151252 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.169169 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.175566 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.188220 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.207482 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.223376 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.245900 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.266918 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.296017 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.321153 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.350430 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.382238 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.420694 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.448132 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.494140 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.525524 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.580111 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.611089 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.676575 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.706020 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.747556 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.763176 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:16:52.813221 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.858844 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.927345 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:52.978796 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:52.996083 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:53.051213 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.106158 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.151943 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:53.186929 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.242822 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.328328 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.388479 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:53.397042 15308 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:53.397246 15308 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:53.397890 15308 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:53.399382 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:53.399549 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:53.399698 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:53.399963 15308 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:16:53.409406 15308 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:53.409585 15308 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:53.410151 15308 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:53.411407 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:53.411587 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:53.411734 14686 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:53.411981 15308 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:53.481158 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.547726 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.644332 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.710008 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.815485 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:53.882865 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:53.982066 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:53.990839 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.069768 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.176358 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.200070 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:54.266192 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.370680 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.463552 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.575384 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.670645 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.725399 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:16:54.790416 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:54.891742 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:54.960929 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:55.013926 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.081681 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:16:55.118355 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:55.213317 15308 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:55.213547 15308 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:55.214339 15308 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:55.215678 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:55.215880 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:55.215981 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:55.216219 15308 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:55.243132 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.270470 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:55.340850 15308 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:55.341092 15308 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:55.341922 15308 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:55.342506 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240502 08:16:55.343318 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:55.343505 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:55.343591 14686 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:55.343858 15308 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:55.348457 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.484316 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.596370 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.735350 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:55.847479 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:55.962586 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:55.996932 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.111066 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.267493 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.290069 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:56.380668 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.547395 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.659157 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:56.838001 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:56.910022 15319 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:56.910189 15319 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:56.910820 15319 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:56.912303 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:56.912436 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:16:56.912564 14686 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:56.912777 15319 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:56.949471 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:56.959335 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:57.134882 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:57.143222 15319 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:57.143426 15319 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:57.144140 15319 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:57.145606 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:57.145741 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:57.145817 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:57.146054 15319 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:57.249953 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:57.309023 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:16:57.340761 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:16:57.444952 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:57.559599 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:57.712618 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:16:57.762701 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:57.873004 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:57.953481 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:58.085060 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:58.198591 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:58.418699 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:58.419880 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:58.534540 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:58.760246 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:58.833125 15319 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:58.833359 15319 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:16:58.833963 15319 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:58.835584 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:58.835799 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:58.835907 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:58.836161 15319 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:58.881558 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:58.960994 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:59.108433 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:16:59.220319 15319 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:16:59.220490 15319 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:16:59.221069 15319 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:16:59.222509 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:59.222703 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:16:59.222836 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:16:59.223049 15319 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:16:59.237422 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:59.467646 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:59.474440 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:16:59.599164 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:59.840407 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:16:59.852807 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:16:59.962481 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:16:59.970829 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:00.214912 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:00.350960 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:00.478053 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:00.596065 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:00.715441 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:17:00.716326 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:17:00.738483 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:00.894021 15319 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:00.894212 15319 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:17:00.894857 15319 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:00.896328 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:00.896517 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:00.896672 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:00.896939 15319 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:17:00.972353 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:00.994302 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:01.135253 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:01.394556 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:01.532232 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:01.537884 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:01.809813 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:01.953976 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:01.959445 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:02.235914 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:02.370335 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:02.521168 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:02.664825 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:02.670315 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:02.956918 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:02.959578 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:03.108886 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:03.240165 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:17:03.404513 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:03.557438 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:03.698539 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:03.859944 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:03.969585 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:04.012338 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:04.171502 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:04.483264 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:04.525993 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:04.779850 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:04.799729 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:04.957811 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:04.971061 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:05.444756 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:05.444808 14712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:05.765877 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 08:17:05.766772 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:05.776873 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:05.923372 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:05.938184 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:05.944497 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:05.959280 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:17:06.022394 15319 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:06.022578 15319 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:06.023280 15319 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:06.024884 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:06.025089 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:06.025244 14686 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:06.025511 15319 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:06.110085 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:06.436401 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:06.618661 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:06.944222 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:06.960534 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:06.965504 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:07.112811 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:17:07.131917 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:07.465449 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:07.658224 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:07.973337 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:07.993223 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:07.994639 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:08.186120 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:08.280335 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:17:08.534503 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:08.730906 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:08.970979 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:09.084424 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:09.084980 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:09.233224 15330 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:09.233412 15330 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:09.234072 15330 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:09.235340 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:09.235489 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:09.235674 14686 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:09.235940 15330 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:09.280059 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:09.583524 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:17:09.641413 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:09.835422 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:09.962360 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:10.204773 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:10.207410 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:10.403314 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:10.637910 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:17:10.776829 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:10.982573 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:10.983213 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:11.028097 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 08:17:11.361627 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:11.362679 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:11.570331 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:11.737334 15335 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:11.737604 15335 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:17:11.738497 15335 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:11.739975 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:11.740533 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:11.740679 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:11.740945 15335 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:11.950644 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:11.969553 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:12.094122 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:17:12.163760 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:12.364454 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:12.551560 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:12.770154 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:12.953835 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:17:12.970404 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:13.165966 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:13.372254 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:13.385427 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:13.785465 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:13.969808 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:14.007418 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:14.413606 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:14.427655 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:14.639428 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:14.763548 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:17:14.972520 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:15.056082 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:15.196422 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:17:15.281476 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:15.482806 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:15.698680 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:15.933367 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:15.973234 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:16.045466 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 08:17:16.354770 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:16.575222 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:16.588124 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:16.967155 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:17.019389 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:17.252822 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:17.315515 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:17.478364 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:17:17.689673 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:17.701988 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:17.927795 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:17.966835 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:17:18.346312 15335 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:18.346485 15335 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:18.347139 15335 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:18.348662 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:18.348951 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:18.349074 14686 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:18.349321 15335 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:18.374614 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:18.611069 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:18.831885 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:18.966907 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:19.064471 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:19.301826 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:19.761236 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:19.792039 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:19.823815 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:17:19.967582 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:19.994969 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:20.007421 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:20.469476 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:20.718947 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:20.982407 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 08:17:21.087145 15344 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:21.087381 15344 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
I20240502 08:17:21.088263 15344 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:21.088969 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 08:17:21.090118 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:21.090270 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:21.090364 14686 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:21.090626 15344 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:21.188985 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:21.201828 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:21.442780 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:21.918974 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:21.969280 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:22.174651 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:22.324394 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:17:22.408080 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:22.442307 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:17:22.654232 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:22.916779 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:22.966379 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:23.397498 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:23.416399 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:23.667505 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:23.967384 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:24.152787 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:24.422878 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:24.447774 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:24.542658 15344 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:24.542852 15344 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:24.543493 15344 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:24.544799 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:24.550653 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:24.550765 14686 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:24.550998 15344 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:24.586638 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:17:24.913110 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:24.947283 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:17:24.994287 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:25.188488 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:25.474865 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:25.688094 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:25.962895 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:25.966816 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:26.471519 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:26.488713 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:26.507212 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 08:17:26.750759 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:26.964453 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:27.121822 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:17:27.261500 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:27.373709 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:17:27.529222 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:27.545996 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:28.003574 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:28.065063 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:28.346902 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:28.603298 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:28.763934 15344 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:28.764099 15344 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:28.764767 15344 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:28.766220 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:28.766409 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:28.766491 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:28.766721 15344 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:28.874223 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:28.978338 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:29.159678 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:29.628862 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:17:29.691603 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:29.706305 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:29.775583 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:17:29.970773 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:29.981328 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:30.513512 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:30.790235 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:30.811204 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:30.990526 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:31.351567 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:31.546290 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 08:17:31.648495 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:31.913056 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:31.933403 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:17:31.975943 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:32.080756 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:32.197804 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:32.498829 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:32.971316 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:33.054759 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:33.069327 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:33.355288 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:33.915813 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:33.968442 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:34.204780 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:34.221423 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:34.603605 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:17:34.721040 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:17:34.787683 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:34.978869 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:35.096127 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:35.373395 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:35.666405 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:35.973544 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:35.980779 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:36.555537 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:36.575497 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:36.820600 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:36.874558 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:36.979869 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:37.079314 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:17:37.230963 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:17:37.455240 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:37.756848 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:37.770542 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:37.971797 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:38.361230 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:38.684569 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:38.969952 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:38.976033 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:39.275758 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:39.474979 15350 raft_consensus.cc:482] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:39.475157 15350 raft_consensus.cc:504] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } } peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } }
I20240502 08:17:39.475771 15350 leader_election.cc:288] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:39.477245 14686 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:39.477380 14685 leader_election.cc:334] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:39.477492 14685 leader_election.cc:302] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:39.477733 15350 raft_consensus.cc:2726] T 301b0b861ed745d4b5c46cad4d7673d9 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:39.603338 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:39.747573 15350 raft_consensus.cc:482] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5497377f4ed44a4284f486249e37c22d)
I20240502 08:17:39.747773 15350 raft_consensus.cc:504] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5497377f4ed44a4284f486249e37c22d" member_type: VOTER last_known_addr { host: "127.14.57.2" port: 39821 } } peers { permanent_uuid: "be64d0fa0ce74ef082a843b34d73d96b" member_type: VOTER last_known_addr { host: "127.14.57.3" port: 44131 } } peers { permanent_uuid: "2fc6868d97f74b6c932851433570b3ba" member_type: VOTER last_known_addr { host: "127.14.57.1" port: 44973 } }
W20240502 08:17:39.748364 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:17:39.748426 15350 leader_election.cc:288] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821), be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131)
W20240502 08:17:39.749672 14686 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:39.749887 14685 leader_election.cc:334] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:39.749969 14685 leader_election.cc:302] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fc6868d97f74b6c932851433570b3ba; no voters: 5497377f4ed44a4284f486249e37c22d, be64d0fa0ce74ef082a843b34d73d96b
I20240502 08:17:39.750212 15350 raft_consensus.cc:2726] T 04285e042f0646a99490ef090adbb766 P 2fc6868d97f74b6c932851433570b3ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:17:39.880334 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:17:39.981184 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:40.197834 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:40.220160 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:40.531658 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:40.996591 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:41.134202 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:41.448256 15067 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
W20240502 08:17:41.470017 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:41.938706 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111) [suppressed 23 similar messages]
I20240502 08:17:41.981107 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:42.079834 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:42.226172 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:17:42.419355 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:42.446036 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:17:42.462507 15067 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111)
I20240502 08:17:42.983894 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:43.030869 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:43.377235 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:43.667959 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:43.979115 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:43.988703 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:44.340695 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:44.708734 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:44.939338 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:17:44.964303 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:44.977564 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:45.105983 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:17:45.315764 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:45.943893 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:45.969069 15066 meta_cache.cc:302] tablet 04285e042f0646a99490ef090adbb766: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
I20240502 08:17:45.971199 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:46.298491 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:46.936371 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:46.971655 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:47.006433 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:47.170500 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:47.171406 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:17:47.287587 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:47.307498 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:17:47.930764 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:47.974375 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:48.009929 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:48.287281 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:48.934927 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:48.980037 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:49.014860 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:49.297160 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:49.636994 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:17:49.804781 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:17:49.949401 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:49.976240 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:50.029876 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:50.317025 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:50.975508 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:50.993326 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:51.055953 15066 meta_cache.cc:302] tablet 301b0b861ed745d4b5c46cad4d7673d9: replica 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821) has failed: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111)
W20240502 08:17:51.343895 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
I20240502 08:17:51.973278 15106 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 08:17:52.001367 14685 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer 5497377f4ed44a4284f486249e37c22d (127.14.57.2:39821): Couldn't send request to peer 5497377f4ed44a4284f486249e37c22d. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.2:39821: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:17:52.007371 14711 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58438: Illegal state: replica 2fc6868d97f74b6c932851433570b3ba is not leader of this config: current role FOLLOWER
W20240502 08:17:52.031560 15067 batcher.cc:441] Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.032420 14564 client-test-util.cc:49] 34 failed ops. First 10 errors follow
I20240502 08:17:52.032553 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-9011889649771282432, int64 link_to=120015187475955712, int64 insert_ts=1714637811899501 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.032793 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-7556968565357412351, int64 link_to=-7716890013575610367, int64 insert_ts=1714637811899711 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.032887 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-6102047480943542270, int64 link_to=2893089596570664962, int64 insert_ts=1714637811899792 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.032984 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-4647126396529672189, int64 link_to=-4943815604480901117, int64 insert_ts=1714637811899845 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033052 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-7839613183622184953, int64 link_to=722348401184473095, int64 insert_ts=1714637811899978 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033154 14564 client-test-util.cc:55] Op INSERT int64 rand_key=3050443799706664968, int64 link_to=-6994541612391137272, int64 insert_ts=1714637811900008 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033253 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-141761512409137140, int64 link_to=-1328236869237342196, int64 insert_ts=1714637811900131 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033335 14564 client-test-util.cc:55] Op INSERT int64 rand_key=1313159572004732941, int64 link_to=-9165001332800618483, int64 insert_ts=1714637811900158 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033401 14564 client-test-util.cc:55] Op INSERT int64 rand_key=2767799181441892366, int64 link_to=1444556064880656398, int64 insert_ts=1714637811900186 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:52.033478 14564 client-test-util.cc:55] Op INSERT int64 rand_key=-3333966824524939248, int64 link_to=4337927136428032016, int64 insert_ts=1714637811900256 had status Timed out: Failed to write batch of 34 ops to tablet 04285e042f0646a99490ef090adbb766 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 04285e042f0646a99490ef090adbb766, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
F20240502 08:17:52.033560 14564 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714637872 (unix time) try "date -d @1714637872" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000038e4) received by PID 14564 (TID 0x7f8eea15ad80) from PID 14564; stack trace: ***
@ 0x7f8ef1b67980 (unknown) at ??:0
@ 0x7f8eeac12fb7 gsignal at ??:0
@ 0x7f8eeac14921 abort at ??:0
@ 0x7f8eed93bdcd google::LogMessage::Fail() at ??:0
@ 0x7f8eed93fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8eed93b7cc google::LogMessage::Flush() at ??:0
@ 0x7f8eed93cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8efd3d1efd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed8c2 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test-util.h:?
@ 0x3d6588 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8889 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7f8eef7d60ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f8eef7cabdb testing::Test::Run() at ??:0
@ 0x7f8eef7cad9d testing::TestInfo::Run() at ??:0
@ 0x7f8eef7cb377 testing::TestSuite::Run() at ??:0
@ 0x7f8eef7cb77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f8eef7d660d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f8eef7cae63 testing::UnitTest::Run() at ??:0
@ 0x7f8f044b53c2 RUN_ALL_TESTS() at ??:0
@ 0x7f8f044b38d9 main at ??:0
@ 0x7f8eeabf5bf7 __libc_start_main at ??:0
@ 0x2ebf5a _start at ??:?
W20240502 08:17:52.094412 14686 consensus_peers.cc:485] T ae01fc8e080f4404831074653d1f293c P 2fc6868d97f74b6c932851433570b3ba -> Peer be64d0fa0ce74ef082a843b34d73d96b (127.14.57.3:44131): Couldn't send request to peer be64d0fa0ce74ef082a843b34d73d96b. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.3:44131: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.