[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 13:59:39.666842 25469 test_util.cc:268] Using random seed: 60921212
I20240502 13:59:39.678858 25469 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 13:59:39.678984 25469 linked_list-test.cc:101] --------------
I20240502 13:59:39.679006 25469 linked_list-test.cc:102] 50 chains
I20240502 13:59:39.679025 25469 linked_list-test.cc:103] 3 tablets
I20240502 13:59:39.679042 25469 linked_list-test.cc:104] Mutations on
I20240502 13:59:39.679071 25469 linked_list-test.cc:105] --------------
I20240502 13:59:39.679181 25469 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:39.679214 25469 ts_itest-base.cc:113] --------------
I20240502 13:59:39.679234 25469 ts_itest-base.cc:114] 3 tablet servers
I20240502 13:59:39.679256 25469 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:39.679283 25469 ts_itest-base.cc:116] --------------
2024-05-02T13:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:39Z Disabled control of system clock
I20240502 13:59:39.692453 25469 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXhC88E/build/debug/bin/kudu
/tmp/dist-test-taskXhC88E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.223.126:42041
--webserver_interface=127.24.223.126
--webserver_port=0
--builtin_ntp_servers=127.24.223.84:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.223.126:42041
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 13:59:39.826368 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.826736 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.826799 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.831341 25481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:39.831461 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.831486 25481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:39.831507 25481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:39.836632 25481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.223.84:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.223.126:42041
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.223.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.24.223.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.25481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.838057 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.839632 25481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.849678 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.850231 25481 server_base.cc:1034] running on GCE node
W20240502 13:59:39.850539 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.851258 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.852018 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.853164 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.854468 25481 hybrid_clock.cc:648] HybridClock initialized: now 1714658379854465 us; error 135 us; skew 500 ppm
I20240502 13:59:39.861593 25481 webserver.cc:460] Webserver started at http://127.24.223.126:43387/ using document root <none> and password file <none>
I20240502 13:59:39.862288 25481 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.862366 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.862581 25481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.864583 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data/instance:
uuid: "4ef313267145411da8448b2e82167c52"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-d0w1"
I20240502 13:59:39.865166 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal/instance:
uuid: "4ef313267145411da8448b2e82167c52"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-d0w1"
I20240502 13:59:39.869661 25481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 13:59:39.873519 25495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.875108 25481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:39.875298 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal
uuid: "4ef313267145411da8448b2e82167c52"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-d0w1"
I20240502 13:59:39.875444 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.895538 25481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.896320 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.896497 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.904517 25481 rpc_server.cc:307] RPC server started. Bound to: 127.24.223.126:42041
I20240502 13:59:39.904521 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.223.126:42041 every 8 connection(s)
I20240502 13:59:39.905834 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/data/info.pb
I20240502 13:59:39.908870 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.910807 25469 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXhC88E/build/debug/bin/kudu as pid 25481
I20240502 13:59:39.911044 25469 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/master-0/wal/instance
I20240502 13:59:39.916023 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52: Bootstrap starting.
I20240502 13:59:39.918548 25548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.919562 25548 log.cc:824] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:39.921707 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52: No bootstrap required, opened a new log
I20240502 13:59:39.924996 25548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } }
I20240502 13:59:39.925266 25548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.925339 25548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef313267145411da8448b2e82167c52, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.925900 25548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } }
I20240502 13:59:39.926095 25548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.926209 25548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.926338 25548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.927480 25548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } }
I20240502 13:59:39.927886 25548 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef313267145411da8448b2e82167c52; no voters:
I20240502 13:59:39.928201 25548 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.928647 25553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.928826 25553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef313267145411da8448b2e82167c52, State: Running, Role: LEADER
I20240502 13:59:39.929456 25553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } }
I20240502 13:59:39.929549 25548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:39.930742 25553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef313267145411da8448b2e82167c52. Latest consensus state: current_term: 1 leader_uuid: "4ef313267145411da8448b2e82167c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } } }
I20240502 13:59:39.930866 25553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:39.931110 25554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef313267145411da8448b2e82167c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef313267145411da8448b2e82167c52" member_type: VOTER last_known_addr { host: "127.24.223.126" port: 42041 } } }
I20240502 13:59:39.931221 25554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:39.931278 25562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:39.934732 25562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:39.941831 25562 catalog_manager.cc:1347] Generated new cluster ID: 73fb666fa444410396e3ed76d49c9902
I20240502 13:59:39.941955 25562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:39.969602 25562 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:39.970674 25562 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:39.981299 25562 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4ef313267145411da8448b2e82167c52: Generated new TSK 0
I20240502 13:59:39.982611 25562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:39.989274 25469 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXhC88E/build/debug/bin/kudu
/tmp/dist-test-taskXhC88E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.223.65:0
--local_ip_for_outbound_sockets=127.24.223.65
--webserver_interface=127.24.223.65
--webserver_port=0
--tserver_master_addrs=127.24.223.126:42041
--builtin_ntp_servers=127.24.223.84:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 13:59:40.125533 25572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.125836 25572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.125902 25572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.129416 25572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.129544 25572 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 13:59:40.129560 25572 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 13:59:40.129581 25572 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 13:59:40.129630 25572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.223.65
I20240502 13:59:40.133935 25572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.223.84:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.223.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.24.223.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.24.223.126:42041
--never_fsync=true
--heap_profile_path=/tmp/kudu.25572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.223.65
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.135097 25572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.136763 25572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.148145 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.148593 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.149325 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.150111 25572 server_base.cc:1034] running on GCE node
I20240502 13:59:40.150642 25572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.151400 25572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.152628 25572 hybrid_clock.cc:648] HybridClock initialized: now 1714658380152603 us; error 54 us; skew 500 ppm
I20240502 13:59:40.155745 25572 webserver.cc:460] Webserver started at http://127.24.223.65:36471/ using document root <none> and password file <none>
I20240502 13:59:40.156586 25572 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.156778 25572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.157138 25572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.159286 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data/instance:
uuid: "f4eb887a0f284e20a3873c688d6cb667"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.159996 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal/instance:
uuid: "f4eb887a0f284e20a3873c688d6cb667"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.164204 25572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 13:59:40.167819 25586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.169468 25572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 13:59:40.169637 25572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal
uuid: "f4eb887a0f284e20a3873c688d6cb667"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.169764 25572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.192441 25572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.193244 25572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.193440 25572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.194542 25572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.195968 25572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.196048 25572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.196125 25572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.196157 25572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.207834 25572 rpc_server.cc:307] RPC server started. Bound to: 127.24.223.65:37497
I20240502 13:59:40.207873 25699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.223.65:37497 every 8 connection(s)
I20240502 13:59:40.208932 25572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/data/info.pb
I20240502 13:59:40.218465 25469 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXhC88E/build/debug/bin/kudu as pid 25572
I20240502 13:59:40.218644 25469 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal/instance
I20240502 13:59:40.219719 25700 heartbeater.cc:347] Connected to a master server at 127.24.223.126:42041
I20240502 13:59:40.220019 25700 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.220793 25700 heartbeater.cc:511] Master 127.24.223.126:42041 requested a full tablet report, sending...
I20240502 13:59:40.221432 25469 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXhC88E/build/debug/bin/kudu
/tmp/dist-test-taskXhC88E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.223.66:0
--local_ip_for_outbound_sockets=127.24.223.66
--webserver_interface=127.24.223.66
--webserver_port=0
--tserver_master_addrs=127.24.223.126:42041
--builtin_ntp_servers=127.24.223.84:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 13:59:40.222560 25507 ts_manager.cc:194] Registered new tserver with Master: f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497)
I20240502 13:59:40.223908 25507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.223.65:52733
W20240502 13:59:40.345522 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.345816 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.345882 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.349485 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.349618 25703 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 13:59:40.349637 25703 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 13:59:40.349658 25703 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 13:59:40.349710 25703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.223.66
I20240502 13:59:40.353962 25703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.223.84:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.223.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.24.223.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.24.223.126:42041
--never_fsync=true
--heap_profile_path=/tmp/kudu.25703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.223.66
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.355262 25703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.357461 25703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.369728 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.370359 25711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.371137 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.372087 25703 server_base.cc:1034] running on GCE node
I20240502 13:59:40.372574 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.373299 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.374518 25703 hybrid_clock.cc:648] HybridClock initialized: now 1714658380374495 us; error 38 us; skew 500 ppm
I20240502 13:59:40.377274 25703 webserver.cc:460] Webserver started at http://127.24.223.66:44105/ using document root <none> and password file <none>
I20240502 13:59:40.377929 25703 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.377995 25703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.378176 25703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.380198 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data/instance:
uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.380821 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal/instance:
uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.384999 25703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 13:59:40.387861 25717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.389453 25703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 13:59:40.389602 25703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal
uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.389744 25703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.416149 25703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.417066 25703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.417255 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.417992 25703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.419111 25703 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.419167 25703 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.419216 25703 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.419260 25703 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.429268 25703 rpc_server.cc:307] RPC server started. Bound to: 127.24.223.66:36681
I20240502 13:59:40.429299 25830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.223.66:36681 every 8 connection(s)
I20240502 13:59:40.430425 25703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/data/info.pb
I20240502 13:59:40.439230 25469 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXhC88E/build/debug/bin/kudu as pid 25703
I20240502 13:59:40.439371 25469 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal/instance
I20240502 13:59:40.441365 25831 heartbeater.cc:347] Connected to a master server at 127.24.223.126:42041
I20240502 13:59:40.441677 25831 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.441838 25469 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXhC88E/build/debug/bin/kudu
/tmp/dist-test-taskXhC88E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.223.67:0
--local_ip_for_outbound_sockets=127.24.223.67
--webserver_interface=127.24.223.67
--webserver_port=0
--tserver_master_addrs=127.24.223.126:42041
--builtin_ntp_servers=127.24.223.84:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 13:59:40.442332 25831 heartbeater.cc:511] Master 127.24.223.126:42041 requested a full tablet report, sending...
I20240502 13:59:40.443548 25507 ts_manager.cc:194] Registered new tserver with Master: 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.444341 25507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.223.66:42663
W20240502 13:59:40.569097 25834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.569406 25834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.569473 25834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.572901 25834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.573021 25834 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 13:59:40.573038 25834 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 13:59:40.573060 25834 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 13:59:40.573105 25834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.223.67
I20240502 13:59:40.577560 25834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.223.84:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.223.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.24.223.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.24.223.126:42041
--never_fsync=true
--heap_profile_path=/tmp/kudu.25834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.223.67
--log_dir=/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.578661 25834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.580025 25834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.589989 25840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.590546 25842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.591897 25839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.592697 25834 server_base.cc:1034] running on GCE node
I20240502 13:59:40.593294 25834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.593994 25834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.595165 25834 hybrid_clock.cc:648] HybridClock initialized: now 1714658380595106 us; error 78 us; skew 500 ppm
I20240502 13:59:40.597589 25834 webserver.cc:460] Webserver started at http://127.24.223.67:41307/ using document root <none> and password file <none>
I20240502 13:59:40.598243 25834 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.598330 25834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.598553 25834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.600556 25834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data/instance:
uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.601135 25834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal/instance:
uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.605010 25834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 13:59:40.607704 25848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.608901 25834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:40.609040 25834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal
uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-d0w1"
I20240502 13:59:40.609179 25834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.622110 25834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.622872 25834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.623085 25834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.623692 25834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.624706 25834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.624759 25834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.624823 25834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.624862 25834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.635591 25834 rpc_server.cc:307] RPC server started. Bound to: 127.24.223.67:39315
I20240502 13:59:40.635622 25961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.223.67:39315 every 8 connection(s)
I20240502 13:59:40.636695 25834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/data/info.pb
I20240502 13:59:40.639906 25469 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXhC88E/build/debug/bin/kudu as pid 25834
I20240502 13:59:40.640054 25469 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal/instance
I20240502 13:59:40.648773 25962 heartbeater.cc:347] Connected to a master server at 127.24.223.126:42041
I20240502 13:59:40.649070 25962 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.649753 25962 heartbeater.cc:511] Master 127.24.223.126:42041 requested a full tablet report, sending...
I20240502 13:59:40.651135 25507 ts_manager.cc:194] Registered new tserver with Master: a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 13:59:40.651741 25507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.223.67:46641
I20240502 13:59:40.656500 25469 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 13:59:40.675151 25507 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49018:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 13:59:40.678761 25507 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 13:59:40.700533 25896 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd9917be1545c996d33376fee706ce (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 13:59:40.701174 25633 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8027e5b0246f79e416a16331707ed (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 13:59:40.701359 25765 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd9917be1545c996d33376fee706ce (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 13:59:40.701177 25763 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 13:59:40.702193 25896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd9917be1545c996d33376fee706ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.702293 25633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8027e5b0246f79e416a16331707ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.701177 25764 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8027e5b0246f79e416a16331707ed (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 13:59:40.702509 25763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.702837 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd9917be1545c996d33376fee706ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.703310 25632 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 13:59:40.703595 25632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.703689 25764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8027e5b0246f79e416a16331707ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.705214 25634 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd9917be1545c996d33376fee706ce (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 13:59:40.705549 25634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd9917be1545c996d33376fee706ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.707139 25894 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 13:59:40.707485 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa3ff04e7e48bea816ec91e7e1bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.709867 25895 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8027e5b0246f79e416a16331707ed (DEFAULT_TABLE table=TestTable [id=6fd7b5bd48cf4df1a4595b806b745339]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 13:59:40.710204 25895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8027e5b0246f79e416a16331707ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.711633 25979 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: Bootstrap starting.
I20240502 13:59:40.714284 25980 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: Bootstrap starting.
I20240502 13:59:40.714977 25979 tablet_bootstrap.cc:654] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.715480 25981 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Bootstrap starting.
I20240502 13:59:40.716384 25979 log.cc:824] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:40.716574 25980 tablet_bootstrap.cc:654] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.717396 25980 log.cc:824] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:40.717708 25981 tablet_bootstrap.cc:654] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.718703 25979 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: No bootstrap required, opened a new log
I20240502 13:59:40.718842 25981 log.cc:824] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:40.718926 25979 ts_tablet_manager.cc:1367] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 13:59:40.720713 25981 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: No bootstrap required, opened a new log
I20240502 13:59:40.720749 25980 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: No bootstrap required, opened a new log
I20240502 13:59:40.720955 25981 ts_tablet_manager.cc:1367] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 13:59:40.720978 25980 ts_tablet_manager.cc:1367] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 13:59:40.722726 25979 raft_consensus.cc:348] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.723143 25979 raft_consensus.cc:374] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.723244 25979 raft_consensus.cc:729] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef6db1b6fa42c7b3078ef12adb4b18, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.723846 25979 consensus_queue.cc:260] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.724603 25981 raft_consensus.cc:348] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.724857 25981 raft_consensus.cc:374] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.724889 25831 heartbeater.cc:503] Master 127.24.223.126:42041 was elected leader, sending a full tablet report...
I20240502 13:59:40.724949 25981 raft_consensus.cc:729] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e1f3276aa441ffba0ab2cdb7e7cbdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.725013 25980 raft_consensus.cc:348] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.725304 25980 raft_consensus.cc:374] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.725376 25980 raft_consensus.cc:729] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eb887a0f284e20a3873c688d6cb667, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.725414 25981 consensus_queue.cc:260] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.725853 25980 consensus_queue.cc:260] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.726152 25962 heartbeater.cc:503] Master 127.24.223.126:42041 was elected leader, sending a full tablet report...
I20240502 13:59:40.726401 25981 ts_tablet_manager.cc:1398] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 13:59:40.726583 25700 heartbeater.cc:503] Master 127.24.223.126:42041 was elected leader, sending a full tablet report...
I20240502 13:59:40.726750 25981 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Bootstrap starting.
I20240502 13:59:40.726763 25980 ts_tablet_manager.cc:1398] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240502 13:59:40.727639 25980 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Bootstrap starting.
I20240502 13:59:40.727860 25979 ts_tablet_manager.cc:1398] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 13:59:40.728188 25979 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Bootstrap starting.
I20240502 13:59:40.728839 25980 tablet_bootstrap.cc:654] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.729357 25979 tablet_bootstrap.cc:654] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.730382 25981 tablet_bootstrap.cc:654] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.730966 25979 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: No bootstrap required, opened a new log
I20240502 13:59:40.731004 25980 tablet_bootstrap.cc:492] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: No bootstrap required, opened a new log
I20240502 13:59:40.731068 25979 ts_tablet_manager.cc:1367] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:40.731107 25980 ts_tablet_manager.cc:1367] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 13:59:40.731881 25980 raft_consensus.cc:348] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.731835 25979 raft_consensus.cc:348] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.732074 25979 raft_consensus.cc:374] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.732118 25980 raft_consensus.cc:374] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.732147 25979 raft_consensus.cc:729] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef6db1b6fa42c7b3078ef12adb4b18, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.732165 25980 raft_consensus.cc:729] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eb887a0f284e20a3873c688d6cb667, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.732308 25979 consensus_queue.cc:260] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.732349 25980 consensus_queue.cc:260] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.732499 25981 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf: No bootstrap required, opened a new log
I20240502 13:59:40.732589 25981 ts_tablet_manager.cc:1367] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:40.732640 25980 ts_tablet_manager.cc:1398] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.732648 25979 ts_tablet_manager.cc:1398] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.732834 25980 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667: Bootstrap starting.
I20240502 13:59:40.732867 25979 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18: Bootstrap starting.
I20240502 13:59:40.733227 25981 raft_consensus.cc:348] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.733368 25981 raft_consensus.cc:374] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.733402 25981 raft_consensus.cc:729] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e1f3276aa441ffba0ab2cdb7e7cbdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.733561 25981 consensus_queue.cc:260] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.733856 25981 ts_tablet_manager.cc:1398] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:40.733947 25980 tablet_bootstrap.cc:654] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.734050 25981 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Bootstrap starting.
I20240502 13:59:40.735231 25981 tablet_bootstrap.cc:654] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.736413 25979 tablet_bootstrap.cc:654] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.736644 25980 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667: No bootstrap required, opened a new log
I20240502 13:59:40.736800 25980 ts_tablet_manager.cc:1367] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:40.737378 25980 raft_consensus.cc:348] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.737520 25980 raft_consensus.cc:374] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.737610 25980 raft_consensus.cc:729] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eb887a0f284e20a3873c688d6cb667, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.737778 25980 consensus_queue.cc:260] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.737953 25979 tablet_bootstrap.cc:492] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18: No bootstrap required, opened a new log
I20240502 13:59:40.738052 25979 ts_tablet_manager.cc:1367] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:40.738142 25980 ts_tablet_manager.cc:1398] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.738891 25979 raft_consensus.cc:348] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.739135 25979 raft_consensus.cc:374] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.739189 25979 raft_consensus.cc:729] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef6db1b6fa42c7b3078ef12adb4b18, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.739271 25981 tablet_bootstrap.cc:492] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf: No bootstrap required, opened a new log
I20240502 13:59:40.739400 25981 ts_tablet_manager.cc:1367] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:40.739480 25979 consensus_queue.cc:260] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.739878 25981 raft_consensus.cc:348] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.739961 25979 ts_tablet_manager.cc:1398] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:40.739987 25981 raft_consensus.cc:374] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.740024 25981 raft_consensus.cc:729] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e1f3276aa441ffba0ab2cdb7e7cbdf, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.740170 25981 consensus_queue.cc:260] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.740458 25981 ts_tablet_manager.cc:1398] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:40.745762 25986 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.745991 25986 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.747207 25986 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.751436 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" is_pre_election: true
I20240502 13:59:40.751626 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667" is_pre_election: true
I20240502 13:59:40.751825 25785 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 0.
I20240502 13:59:40.751962 25654 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 0.
I20240502 13:59:40.752286 25852 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf; no voters:
I20240502 13:59:40.752565 25986 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.752655 25986 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.752707 25986 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.753671 25986 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.754045 25986 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 election: Requested vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.754467 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667"
I20240502 13:59:40.754503 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
I20240502 13:59:40.754621 25785 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.754621 25654 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.755658 25785 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 1.
I20240502 13:59:40.755681 25654 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 1.
I20240502 13:59:40.756064 25852 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf; no voters:
I20240502 13:59:40.756317 25986 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.756556 25986 raft_consensus.cc:686] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 LEADER]: Becoming Leader. State: Replica: a0e1f3276aa441ffba0ab2cdb7e7cbdf, State: Running, Role: LEADER
I20240502 13:59:40.756917 25986 consensus_queue.cc:237] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.760522 25512 catalog_manager.cc:5484] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67). New cstate: current_term: 1 leader_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:40.786088 25986 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.786267 25986 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.786763 25986 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.787277 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667" is_pre_election: true
I20240502 13:59:40.787317 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" is_pre_election: true
I20240502 13:59:40.787442 25654 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 0.
I20240502 13:59:40.787498 25785 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 0.
I20240502 13:59:40.787798 25850 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 13:59:40.788005 25986 raft_consensus.cc:2781] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.788067 25986 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.788115 25986 raft_consensus.cc:3037] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.789016 25986 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.789402 25986 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 election: Requested vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.789829 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667"
I20240502 13:59:40.789891 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
I20240502 13:59:40.789970 25654 raft_consensus.cc:3037] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.790050 25785 raft_consensus.cc:3037] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.790839 25654 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 1.
I20240502 13:59:40.790844 25785 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e1f3276aa441ffba0ab2cdb7e7cbdf in term 1.
I20240502 13:59:40.791247 25850 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 13:59:40.791491 25986 raft_consensus.cc:2781] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.791606 25986 raft_consensus.cc:686] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 LEADER]: Becoming Leader. State: Replica: a0e1f3276aa441ffba0ab2cdb7e7cbdf, State: Running, Role: LEADER
I20240502 13:59:40.791800 25986 consensus_queue.cc:237] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 13:59:40.793692 25512 catalog_manager.cc:5484] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67). New cstate: current_term: 1 leader_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } health_report { overall_health: HEALTHY } } }
W20240502 13:59:40.887763 25963 tablet.cc:2369] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:40.905898 25987 raft_consensus.cc:482] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:40.906059 25987 raft_consensus.cc:504] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.907258 25987 leader_election.cc:288] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.912156 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd9917be1545c996d33376fee706ce" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" is_pre_election: true
I20240502 13:59:40.912231 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd9917be1545c996d33376fee706ce" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" is_pre_election: true
I20240502 13:59:40.912348 25785 raft_consensus.cc:2454] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 0.
I20240502 13:59:40.912451 25916 raft_consensus.cc:2454] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 0.
I20240502 13:59:40.912909 25590 leader_election.cc:302] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 13:59:40.913261 25987 raft_consensus.cc:2781] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:40.913352 25987 raft_consensus.cc:482] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:40.913399 25987 raft_consensus.cc:3037] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.914901 25987 raft_consensus.cc:504] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.915529 25987 leader_election.cc:288] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 1 election: Requested vote from peers a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315), 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681)
I20240502 13:59:40.916108 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd9917be1545c996d33376fee706ce" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
I20240502 13:59:40.916141 25916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd9917be1545c996d33376fee706ce" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf"
I20240502 13:59:40.916270 25916 raft_consensus.cc:3037] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.916271 25785 raft_consensus.cc:3037] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.917354 25916 raft_consensus.cc:2454] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 1.
I20240502 13:59:40.917356 25785 raft_consensus.cc:2454] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 1.
I20240502 13:59:40.917879 25587 leader_election.cc:302] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 13:59:40.918133 25987 raft_consensus.cc:2781] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.918383 25987 raft_consensus.cc:686] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [term 1 LEADER]: Becoming Leader. State: Replica: f4eb887a0f284e20a3873c688d6cb667, State: Running, Role: LEADER
I20240502 13:59:40.919111 25987 consensus_queue.cc:237] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } }
I20240502 13:59:40.923389 25512 catalog_manager.cc:5484] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65). New cstate: current_term: 1 leader_uuid: "f4eb887a0f284e20a3873c688d6cb667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:40.931759 25832 tablet.cc:2369] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:40.951710 25469 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 13:59:40.953856 25469 ts_itest-base.cc:233] Waiting for 3 tablets on tserver f4eb887a0f284e20a3873c688d6cb667 to finish bootstrapping
W20240502 13:59:40.960386 25701 tablet.cc:2369] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:40.961109 25469 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 41ef6db1b6fa42c7b3078ef12adb4b18 to finish bootstrapping
I20240502 13:59:40.966856 25469 ts_itest-base.cc:233] Waiting for 3 tablets on tserver a0e1f3276aa441ffba0ab2cdb7e7cbdf to finish bootstrapping
I20240502 13:59:40.973508 26003 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.24.223.66:44105/scans
http://127.24.223.65:36471/stacks
http://127.24.223.67:41307/threadz?group=all
http://127.24.223.126:43387/threadz?group=all
http://127.24.223.67:41307/maintenance-manager
http://127.24.223.67:41307/scans
http://127.24.223.65:36471/tablets
http://127.24.223.126:43387/startup
http://127.24.223.65:36471/mem-trackers
http://127.24.223.67:41307/stacks
http://127.24.223.126:43387/stacks
http://127.24.223.66:44105/threadz?group=all
http://127.24.223.126:43387/metrics
http://127.24.223.67:41307/tablets
http://127.24.223.66:44105/stacks
http://127.24.223.65:36471/maintenance-manager
http://127.24.223.67:41307/metrics
http://127.24.223.65:36471/startup
http://127.24.223.126:43387/ipki-ca-cert
http://127.24.223.65:36471/scans
http://127.24.223.66:44105/maintenance-manager
http://127.24.223.65:36471/threadz?group=all
http://127.24.223.65:36471/transactions?tablet_id=4ffa3ff04e7e48bea816ec91e7e1bd10
http://127.24.223.126:43387/mem-trackers
http://127.24.223.67:41307/startup
http://127.24.223.66:44105/threadz
http://127.24.223.65:36471/metrics
http://127.24.223.67:41307/mem-trackers
http://127.24.223.67:41307/threadz
http://127.24.223.66:44105/metrics
http://127.24.223.66:44105/transactions?tablet_id=4ffa3ff04e7e48bea816ec91e7e1bd10
http://127.24.223.66:44105/tablets
http://127.24.223.126:43387/dump-entities
http://127.24.223.66:44105/mem-trackers
http://127.24.223.126:43387/threadz
http://127.24.223.126:43387/ipki-ca-cert-der
http://127.24.223.66:44105/startup
http://127.24.223.67:41307/transactions?tablet_id=4ffa3ff04e7e48bea816ec91e7e1bd10
http://127.24.223.126:43387/tables
http://127.24.223.65:36471/threadz
http://127.24.223.126:43387/ipki-ca-cert-pem
http://127.24.223.126:43387/masters
http://127.24.223.126:43387/tablet-servers
I20240502 13:59:40.975417 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:40.977602 25469 linked_list-test-util.h:401] Running for: 30.000s
I20240502 13:59:40.977692 25469 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 13:59:40.977710 25469 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 13:59:41.001870 25654 raft_consensus.cc:1261] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Refusing update from remote peer a0e1f3276aa441ffba0ab2cdb7e7cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.003057 25784 raft_consensus.cc:1261] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Refusing update from remote peer a0e1f3276aa441ffba0ab2cdb7e7cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.003185 25995 consensus_queue.cc:1035] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.003607 25916 raft_consensus.cc:1261] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf [term 1 FOLLOWER]: Refusing update from remote peer f4eb887a0f284e20a3873c688d6cb667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.004098 25783 raft_consensus.cc:1261] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Refusing update from remote peer a0e1f3276aa441ffba0ab2cdb7e7cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.004607 25995 consensus_queue.cc:1035] T a4e8027e5b0246f79e416a16331707ed P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.004644 25986 consensus_queue.cc:1035] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.006063 25987 consensus_queue.cc:1035] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.006681 25785 raft_consensus.cc:1261] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Refusing update from remote peer f4eb887a0f284e20a3873c688d6cb667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.007290 25987 consensus_queue.cc:1035] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.008128 25654 raft_consensus.cc:1261] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Refusing update from remote peer a0e1f3276aa441ffba0ab2cdb7e7cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:41.011639 26011 mvcc.cc:205] Tried to move back new op lower bound from 7023240728576835584 to 7023240727724961792. Current Snapshot: MvccSnapshot[applied={T|T < 7023240728576835584}]
I20240502 13:59:41.012940 25995 consensus_queue.cc:1035] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P a0e1f3276aa441ffba0ab2cdb7e7cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:41.015200 26015 mvcc.cc:205] Tried to move back new op lower bound from 7023240728572227584 to 7023240727583199232. Current Snapshot: MvccSnapshot[applied={T|T < 7023240728572227584}]
I20240502 13:59:41.022708 26026 mvcc.cc:205] Tried to move back new op lower bound from 7023240728576561152 to 7023240728248205312. Current Snapshot: MvccSnapshot[applied={T|T < 7023240728576561152}]
I20240502 13:59:41.689636 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:44.071707 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:44.238565 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:44.993999 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:45.220813 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:45.397655 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:45.442770 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:45.629793 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.408s user 0.280s sys 0.000s Metrics: {"bytes_written":143923,"cfile_init":1,"compiler_manager_pool.queue_time_us":35147,"compiler_manager_pool.run_cpu_time_us":266274,"compiler_manager_pool.run_wall_time_us":325244,"dirs.queue_time_us":11872,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":9025,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1985,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7896,"thread_start_us":253,"threads_started":2}
I20240502 13:59:45.631457 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 139848 bytes on disk
I20240502 13:59:45.632318 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 13:59:45.931363 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.532s user 0.284s sys 0.004s Metrics: {"bytes_written":146873,"cfile_init":1,"compiler_manager_pool.queue_time_us":202447,"dirs.queue_time_us":12695,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":5428,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1869,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8063,"thread_start_us":261,"threads_started":2}
I20240502 13:59:45.933332 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 141530 bytes on disk
I20240502 13:59:45.937682 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 13:59:46.216406 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.773s user 0.301s sys 0.004s Metrics: {"bytes_written":147618,"cfile_init":1,"compiler_manager_pool.queue_time_us":26510,"dirs.queue_time_us":18439,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1242,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1881,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8098,"thread_start_us":224,"threads_started":2}
I20240502 13:59:46.217783 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 141805 bytes on disk
I20240502 13:59:46.218763 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 13:59:47.602996 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:47.702409 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:48.468482 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:50.142119 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:50.203545 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:50.230054 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:50.733937 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.504s user 0.245s sys 0.006s Metrics: {"bytes_written":11928,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1406,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":3211,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2104,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8057,"thread_start_us":82,"threads_started":1,"update_count":1836}
I20240502 13:59:50.735307 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 128541 bytes on disk
I20240502 13:59:50.735891 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 13:59:50.736572 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014818
I20240502 13:59:50.773859 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.631s user 0.248s sys 0.000s Metrics: {"bytes_written":11062,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9964,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":9264,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11652,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8130,"thread_start_us":118,"threads_started":1,"update_count":1707}
I20240502 13:59:50.776448 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 130994 bytes on disk
I20240502 13:59:50.777395 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 13:59:50.778172 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014819
I20240502 13:59:50.846554 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:50.848505 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.645s user 0.237s sys 0.012s Metrics: {"bytes_written":11689,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12707,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":3745,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2119,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8061,"thread_start_us":93,"threads_started":1,"update_count":1805}
I20240502 13:59:50.849835 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014818
I20240502 13:59:50.996286 25469 linked_list-test-util.h:417] Sample at HT timestamp: 7023240769474310144 Inserted count: 25200
I20240502 13:59:51.013317 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:51.373371 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.637s user 0.414s sys 0.008s Metrics: {"bytes_written":298175,"cfile_cache_hit":13,"cfile_cache_hit_bytes":95790,"cfile_cache_miss":61,"cfile_cache_miss_bytes":977688,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":16229,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":2188,"drs_written":1,"lbm_read_time_us":2439,"lbm_reads_lt_1ms":93,"lbm_write_time_us":3357,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":526197,"rows_written":16155,"thread_start_us":538,"threads_started":8}
I20240502 13:59:51.374177 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 298288 bytes on disk
I20240502 13:59:51.374704 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 13:59:51.507063 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.643s user 0.418s sys 0.015s Metrics: {"bytes_written":297566,"cfile_cache_hit":13,"cfile_cache_hit_bytes":95232,"cfile_cache_miss":59,"cfile_cache_miss_bytes":976800,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":14718,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":3895,"drs_written":1,"lbm_read_time_us":2403,"lbm_reads_lt_1ms":95,"lbm_write_time_us":3779,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":525790,"rows_written":16124,"thread_start_us":609,"threads_started":8}
I20240502 13:59:51.507984 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 297527 bytes on disk
I20240502 13:59:51.508536 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 13:59:51.589691 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.810s user 0.425s sys 0.003s Metrics: {"bytes_written":296107,"cfile_cache_hit":13,"cfile_cache_hit_bytes":93421,"cfile_cache_miss":59,"cfile_cache_miss_bytes":974627,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":51267,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":3595,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":90,"lbm_write_time_us":2942,"lbm_writes_lt_1ms":51,"mutex_wait_us":367,"num_input_rowsets":2,"peak_mem_usage":525520,"rows_written":16026,"spinlock_wait_cycles":12672,"thread_start_us":624,"threads_started":8}
I20240502 13:59:51.591277 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 295619 bytes on disk
I20240502 13:59:51.591902 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 13:59:52.132504 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:54.263028 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:54.371603 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:55.099359 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:55.120707 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:55.134327 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:55.267951 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 13:59:55.390676 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.291s user 0.252s sys 0.004s Metrics: {"bytes_written":146368,"cfile_init":1,"dirs.queue_time_us":18829,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1778,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7918,"spinlock_wait_cycles":8960,"thread_start_us":100,"threads_started":1}
I20240502 13:59:55.391480 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014707
I20240502 13:59:55.619484 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.485s user 0.227s sys 0.012s Metrics: {"bytes_written":145540,"cfile_init":1,"dirs.queue_time_us":16384,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":21303,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1905,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7857,"thread_start_us":94,"threads_started":1}
I20240502 13:59:55.620462 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 130051 bytes on disk
I20240502 13:59:55.627506 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 13:59:55.628264 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014707
I20240502 13:59:55.905494 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.637s user 0.267s sys 0.018s Metrics: {"bytes_written":9393,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15259,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":8110,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10189,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8092,"thread_start_us":125,"threads_started":1,"update_count":1440}
I20240502 13:59:55.909551 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 132828 bytes on disk
I20240502 13:59:55.910111 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 13:59:55.911345 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014701
I20240502 13:59:56.678311 26112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57942 (ReqId={client: 9eae1dd3eb094a00adb752da065f354a, seq_no=2215, attempt_no=0}) took 1180 ms. Trace:
I20240502 13:59:56.687007 26112 rpcz_store.cc:276] 0502 13:59:55.497715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 13:59:55.499415 (+ 1700us) service_pool.cc:225] Handling call
0502 13:59:56.678291 (+1178876us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 13:59:55.500133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3fdd9917be1545c996d33376fee706ce
0502 13:59:55.500206 (+ 73us) write_op.cc:424] Acquiring schema lock in shared mode
0502 13:59:55.500210 (+ 4us) write_op.cc:427] Acquired schema lock
0502 13:59:55.500211 (+ 1us) tablet.cc:660] Decoding operations
0502 13:59:55.509118 (+ 8907us) write_op.cc:612] Acquiring the partition lock for write op
0502 13:59:55.509133 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0502 13:59:55.509140 (+ 7us) tablet.cc:683] Acquiring locks for 1700 operations
0502 13:59:55.579889 (+ 70749us) tablet.cc:699] Row locks acquired
0502 13:59:55.579893 (+ 4us) write_op.cc:260] PREPARE: finished
0502 13:59:55.579979 (+ 86us) write_op.cc:270] Start()
0502 13:59:55.580022 (+ 43us) write_op.cc:276] Timestamp: P: 1714658395579949 usec, L: 0
0502 13:59:55.580025 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 13:59:55.580346 (+ 321us) log.cc:842] Serialized 19050 byte log entry
0502 13:59:55.594988 (+ 14642us) op_driver.cc:444] REPLICATION: finished
0502 13:59:55.595161 (+ 173us) write_op.cc:301] APPLY: starting
0502 13:59:55.595178 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0502 13:59:56.114736 (+519558us) tablet.cc:1368] finished BulkCheckPresence
0502 13:59:56.114756 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 13:59:56.653774 (+539018us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 13:59:56.654548 (+ 774us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=3400,key_file_lookups=2336,delta_file_lookups=0,mrs_lookups=532
0502 13:59:56.654556 (+ 8us) write_op.cc:312] APPLY: finished
0502 13:59:56.677556 (+ 23000us) log.cc:842] Serialized 12556 byte log entry
0502 13:59:56.677651 (+ 95us) write_op.cc:481] Releasing partition, row and schema locks
0502 13:59:56.678084 (+ 433us) write_op.cc:446] Released schema lock
0502 13:59:56.678193 (+ 109us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"cfile_cache_hit":5719,"cfile_cache_hit_bytes":113106519,"cfile_cache_miss":2,"cfile_cache_miss_bytes":46595,"lbm_read_time_us":56,"lbm_reads_lt_1ms":2,"num_ops":1700,"prepare.queue_time_us":409,"prepare.run_cpu_time_us":17366,"prepare.run_wall_time_us":113299,"raft.queue_time_us":90,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":576,"replication_time_us":14936,"spinlock_wait_cycles":16512}]]}
I20240502 13:59:56.939714 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 1.548s user 0.602s sys 0.004s Metrics: {"bytes_written":7624,"cfile_cache_hit":13,"cfile_cache_hit_bytes":231750,"cfile_cache_miss":73,"cfile_cache_miss_bytes":1382558,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":4021,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":978,"drs_written":1,"lbm_read_time_us":5741,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":104,"lbm_write_time_us":6372,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":66,"num_input_rowsets":2,"peak_mem_usage":771003,"reinsert_count":0,"rows_written":23944,"thread_start_us":494,"threads_started":7,"update_count":1168}
I20240502 13:59:56.940905 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 440078 bytes on disk
I20240502 13:59:56.941496 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 13:59:56.942088 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling MajorDeltaCompactionOp(3fdd9917be1545c996d33376fee706ce): perf score=0.019949
I20240502 13:59:57.017854 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 1.106s user 0.828s sys 0.008s Metrics: {"bytes_written":445449,"cfile_cache_hit":13,"cfile_cache_hit_bytes":186455,"cfile_cache_miss":77,"cfile_cache_miss_bytes":1460685,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":1860,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1880,"drs_written":1,"lbm_read_time_us":4502,"lbm_reads_lt_1ms":109,"lbm_write_time_us":11759,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":63,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":801048,"rows_written":24216,"thread_start_us":611,"threads_started":8}
I20240502 13:59:57.018824 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 455677 bytes on disk
I20240502 13:59:57.019347 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 13:59:57.048990 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: MajorDeltaCompactionOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.107s user 0.057s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":10337,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":65,"lbm_read_time_us":275,"lbm_reads_lt_1ms":20,"lbm_write_time_us":285,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1168}
I20240502 13:59:57.364733 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 1.736s user 0.703s sys 0.020s Metrics: {"bytes_written":8059,"cfile_cache_hit":14,"cfile_cache_hit_bytes":279237,"cfile_cache_miss":72,"cfile_cache_miss_bytes":1339495,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":12046,"dirs.run_cpu_time_us":392,"dirs.run_wall_time_us":7686,"drs_written":1,"lbm_read_time_us":3421,"lbm_reads_lt_1ms":100,"lbm_write_time_us":4688,"lbm_writes_lt_1ms":67,"num_input_rowsets":2,"peak_mem_usage":772070,"reinsert_count":0,"rows_written":24012,"spinlock_wait_cycles":896,"thread_start_us":595,"threads_started":7,"update_count":1236}
I20240502 13:59:57.365931 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 440933 bytes on disk
I20240502 13:59:57.366752 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 13:59:57.370324 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling MajorDeltaCompactionOp(3fdd9917be1545c996d33376fee706ce): perf score=0.021018
I20240502 13:59:57.466310 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: MajorDeltaCompactionOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.095s user 0.073s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":10848,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":353,"lbm_reads_lt_1ms":20,"lbm_write_time_us":312,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1236}
I20240502 13:59:58.232589 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:58.336017 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:59.025439 26036 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 13:59:59.470932 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(a4e8027e5b0246f79e416a16331707ed): perf score=1.000000
I20240502 13:59:59.529850 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(a4e8027e5b0246f79e416a16331707ed): perf score=1.000000
I20240502 13:59:59.556501 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(a4e8027e5b0246f79e416a16331707ed): perf score=1.000000
I20240502 13:59:59.795624 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.323s user 0.220s sys 0.007s Metrics: {"bytes_written":142789,"cfile_init":1,"dirs.queue_time_us":14144,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1922,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7642,"thread_start_us":96,"threads_started":1}
I20240502 13:59:59.796440 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10): perf score=1.000000
I20240502 13:59:59.839677 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.310s user 0.224s sys 0.006s Metrics: {"bytes_written":143050,"cfile_init":1,"dirs.queue_time_us":12799,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1358,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1788,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7663,"thread_start_us":90,"threads_started":1}
I20240502 13:59:59.840564 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed): 142620 bytes on disk
I20240502 13:59:59.841176 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 13:59:59.841794 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10): perf score=1.000000
I20240502 14:00:00.099982 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.258s user 0.214s sys 0.009s Metrics: {"bytes_written":141880,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1099,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1863,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7640}
I20240502 14:00:00.100873 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10): 145493 bytes on disk
I20240502 14:00:00.101408 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 14:00:00.221271 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.665s user 0.244s sys 0.000s Metrics: {"bytes_written":2610,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":22214,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1849,"drs_written":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9315,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7663,"thread_start_us":110,"threads_started":1,"update_count":379}
I20240502 14:00:00.222162 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10): perf score=1.000000
I20240502 14:00:00.360442 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.564s user 0.227s sys 0.004s Metrics: {"bytes_written":141726,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1027,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1739,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7631,"spinlock_wait_cycles":240896}
I20240502 14:00:00.366763 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10): 145361 bytes on disk
I20240502 14:00:00.368464 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:00:00.369194 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed): 142403 bytes on disk
I20240502 14:00:00.378186 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 14:00:00.568070 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.341s user 0.231s sys 0.001s Metrics: {"bytes_written":143987,"cfile_init":1,"dirs.queue_time_us":13827,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2484,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1767,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7750,"thread_start_us":101,"threads_started":1}
I20240502 14:00:00.568930 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10): 146622 bytes on disk
I20240502 14:00:00.569566 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(4ffa3ff04e7e48bea816ec91e7e1bd10) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 14:00:00.570783 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed): 142620 bytes on disk
I20240502 14:00:00.571380 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 14:00:00.571991 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling MajorDeltaCompactionOp(a4e8027e5b0246f79e416a16331707ed): perf score=0.021324
I20240502 14:00:00.602304 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: MajorDeltaCompactionOp(a4e8027e5b0246f79e416a16331707ed) complete. Timing: real 0.030s user 0.027s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3461,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":41,"lbm_read_time_us":258,"lbm_reads_lt_1ms":20,"lbm_write_time_us":295,"lbm_writes_lt_1ms":8,"mutex_wait_us":22,"reinsert_count":0,"thread_start_us":78,"threads_started":1,"update_count":379}
I20240502 14:00:00.853385 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 14:00:00.853837 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 14:00:00.879947 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling FlushMRSOp(3fdd9917be1545c996d33376fee706ce): perf score=1.000000
I20240502 14:00:00.987032 25469 linked_list-test-util.h:417] Sample at HT timestamp: 7023240810420510720 Inserted count: 48650
I20240502 14:00:01.418437 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.564s user 0.298s sys 0.002s Metrics: {"bytes_written":10060,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":10746,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":50391,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1987,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8266,"thread_start_us":81,"threads_started":1,"update_count":1546}
I20240502 14:00:01.419660 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling LogGCOp(3fdd9917be1545c996d33376fee706ce): free 1050357 bytes of WAL
I20240502 14:00:01.420459 25591 log_reader.cc:386] T 3fdd9917be1545c996d33376fee706ce: removed 1 log segments from log reader
I20240502 14:00:01.420544 25591 log.cc:1076] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667: Deleting log segment in path: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-0/wal/wals/3fdd9917be1545c996d33376fee706ce/wal-000000001 (ops 1-741)
I20240502 14:00:01.421144 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: LogGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240502 14:00:01.421794 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014573
I20240502 14:00:01.519891 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.666s user 0.274s sys 0.005s Metrics: {"bytes_written":10052,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9760,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9798,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7994,"thread_start_us":82,"threads_started":1,"update_count":1546}
I20240502 14:00:01.523695 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling LogGCOp(3fdd9917be1545c996d33376fee706ce): free 1049393 bytes of WAL
I20240502 14:00:01.524056 25853 log_reader.cc:386] T 3fdd9917be1545c996d33376fee706ce: removed 1 log segments from log reader
I20240502 14:00:01.524130 25853 log.cc:1076] T 3fdd9917be1545c996d33376fee706ce P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Deleting log segment in path: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-2/wal/wals/3fdd9917be1545c996d33376fee706ce/wal-000000001 (ops 1-741)
I20240502 14:00:01.524670 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: LogGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240502 14:00:01.525138 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 129987 bytes on disk
I20240502 14:00:01.525699 25853 maintenance_manager.cc:643] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 14:00:01.526332 25963 maintenance_manager.cc:419] P a0e1f3276aa441ffba0ab2cdb7e7cbdf: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014571
I20240502 14:00:01.628675 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: FlushMRSOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.748s user 0.256s sys 0.004s Metrics: {"bytes_written":10053,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":14695,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2741,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8198,"thread_start_us":84,"threads_started":1,"update_count":1546}
I20240502 14:00:01.629793 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling LogGCOp(3fdd9917be1545c996d33376fee706ce): free 1048669 bytes of WAL
I20240502 14:00:01.630190 25722 log_reader.cc:386] T 3fdd9917be1545c996d33376fee706ce: removed 1 log segments from log reader
I20240502 14:00:01.630344 25722 log.cc:1076] T 3fdd9917be1545c996d33376fee706ce P 41ef6db1b6fa42c7b3078ef12adb4b18: Deleting log segment in path: /tmp/dist-test-taskXhC88E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714658379654803-25469-0/linked-list-cluster/ts-1/wal/wals/3fdd9917be1545c996d33376fee706ce/wal-000000001 (ops 1-740)
I20240502 14:00:01.631009 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: LogGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {}
I20240502 14:00:01.631703 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce): perf score=1.014573
I20240502 14:00:02.279083 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 14:00:02.415928 26036 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7ffb8a590dcd google::LogMessage::Fail() at ??:0
@ 0x7ffb8a594b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffb8a5907cc google::LogMessage::Flush() at ??:0
@ 0x7ffb8a591f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffb8b9725b1 kudu::operator-() at ??:0
@ 0x7ffb8da55fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7ffb8da3f05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7ffb8da3ef90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7ffb8da159bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7ffb8e641861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7ffb8e64093a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7ffb8e649149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7ffb8950bd09 std::function<>::operator()() at ??:0
@ 0x7ffb89506107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7ffb89505a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7ffb89505123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7ffb8952db89 handle_request at ??:0
@ 0x7ffb8953026b worker_thread at ??:0
@ 0x7ffb8c20f6db start_thread at ??:0
@ 0x7ffb8974771f clone at ??:0
I20240502 14:00:02.672870 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 1.251s user 0.754s sys 0.010s Metrics: {"bytes_written":390,"cfile_cache_hit":14,"cfile_cache_hit_bytes":98126,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2104668,"cfile_init":10,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":11746,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1615,"drs_written":1,"lbm_read_time_us":4272,"lbm_reads_lt_1ms":134,"lbm_write_time_us":6275,"lbm_writes_lt_1ms":80,"num_input_rowsets":2,"peak_mem_usage":1068221,"reinsert_count":0,"rows_written":32210,"thread_start_us":684,"threads_started":8,"update_count":33}
I20240502 14:00:02.673853 25701 maintenance_manager.cc:419] P f4eb887a0f284e20a3873c688d6cb667: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 613100 bytes on disk
I20240502 14:00:02.674444 25591 maintenance_manager.cc:643] P f4eb887a0f284e20a3873c688d6cb667: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4}
W20240502 14:00:02.829149 25965 meta_cache.cc:302] tablet a4e8027e5b0246f79e416a16331707ed: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: recv got EOF from 127.24.223.67:39315 (error 108)
W20240502 14:00:02.834815 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:02.835738 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:02.857312 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.872133 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.883272 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
I20240502 14:00:02.899977 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: CompactRowSetsOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 1.268s user 0.794s sys 0.015s Metrics: {"bytes_written":390,"cfile_cache_hit":13,"cfile_cache_hit_bytes":49153,"cfile_cache_miss":95,"cfile_cache_miss_bytes":2154017,"cfile_init":10,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":1290,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":721,"drs_written":1,"lbm_read_time_us":4237,"lbm_reads_lt_1ms":135,"lbm_write_time_us":18821,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":79,"num_input_rowsets":2,"peak_mem_usage":1068217,"reinsert_count":0,"rows_written":32210,"thread_start_us":500,"threads_started":8,"update_count":33}
I20240502 14:00:02.900940 25832 maintenance_manager.cc:419] P 41ef6db1b6fa42c7b3078ef12adb4b18: Scheduling UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce): 613100 bytes on disk
I20240502 14:00:02.901466 25722 maintenance_manager.cc:643] P 41ef6db1b6fa42c7b3078ef12adb4b18: UndoDeltaBlockGCOp(3fdd9917be1545c996d33376fee706ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 14:00:02.906312 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.910876 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.915793 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.916793 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.920398 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.920578 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:02.921185 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.926087 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:02.933141 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.939882 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.945943 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.947088 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.961359 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.961359 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.967916 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.972429 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:02.998375 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.002768 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.005957 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.014887 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.019353 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.022399 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.050192 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.054801 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.069335 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.069693 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.076467 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.087246 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.109071 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.123478 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.127709 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.148095 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.172052 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.196316 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.198549 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.198549 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.225088 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.244911 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.251456 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.272193 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.287400 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.316112 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.324247 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.351792 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.379588 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.412259 25613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.413245 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.429162 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.444381 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.484333 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.514158 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.520584 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.531590 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.549000 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.595679 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.626345 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.636502 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.663074 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.719556 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.747444 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.763038 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.764395 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.791136 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.803323 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.837172 25965 meta_cache.cc:302] tablet a4e8027e5b0246f79e416a16331707ed: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:03.850667 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.878288 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:03.894840 26258 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.895062 26258 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
W20240502 14:00:03.896191 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
I20240502 14:00:03.896394 26258 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:03.897377 25718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:03.898432 25718 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.900295 26209 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.900458 26209 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.901137 26209 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:03.901494 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667" is_pre_election: true
I20240502 14:00:03.901516 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" is_pre_election: true
I20240502 14:00:03.901662 25654 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef6db1b6fa42c7b3078ef12adb4b18 in term 1.
I20240502 14:00:03.901664 25785 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 1.
I20240502 14:00:03.902021 25590 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 14:00:03.902222 26209 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:00:03.902165 25719 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:03.902278 26209 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.902330 26209 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:03.902459 26258 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:00:03.902513 26258 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.902547 26258 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:03.903182 25587 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.903712 26258 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.903721 26209 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.904143 26258 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: Requested vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:03.904177 26209 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 election: Requested vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:03.904582 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667"
I20240502 14:00:03.904773 25654 raft_consensus.cc:2379] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 41ef6db1b6fa42c7b3078ef12adb4b18 in current term 2: Already voted for candidate f4eb887a0f284e20a3873c688d6cb667 in this term.
I20240502 14:00:03.904745 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
I20240502 14:00:03.904966 25785 raft_consensus.cc:2379] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f4eb887a0f284e20a3873c688d6cb667 in current term 2: Already voted for candidate 41ef6db1b6fa42c7b3078ef12adb4b18 in this term.
W20240502 14:00:03.905344 25718 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.905437 25718 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18; no voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf, f4eb887a0f284e20a3873c688d6cb667
I20240502 14:00:03.905680 26258 raft_consensus.cc:2726] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 14:00:03.906603 25587 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.906698 25587 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:03.906881 26209 raft_consensus.cc:2726] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 14:00:03.923017 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:03.932669 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:03.941742 26258 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.941891 26258 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.942389 26258 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:03.943011 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667" is_pre_election: true
I20240502 14:00:03.943218 25654 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef6db1b6fa42c7b3078ef12adb4b18 in term 1.
I20240502 14:00:03.943593 25719 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 14:00:03.943876 26258 raft_consensus.cc:2781] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:00:03.943939 26258 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0e1f3276aa441ffba0ab2cdb7e7cbdf)
I20240502 14:00:03.943977 26258 raft_consensus.cc:3037] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:03.944296 25718 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.945197 26258 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.945590 26258 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: Requested vote from peers f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65:37497), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:03.946121 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa3ff04e7e48bea816ec91e7e1bd10" candidate_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" candidate_term: 2 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "f4eb887a0f284e20a3873c688d6cb667"
I20240502 14:00:03.946282 25654 raft_consensus.cc:3037] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:03.946763 25718 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:03.947419 25654 raft_consensus.cc:2454] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef6db1b6fa42c7b3078ef12adb4b18 in term 2.
I20240502 14:00:03.947772 25719 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:03.948139 26258 raft_consensus.cc:2781] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:03.948421 26258 raft_consensus.cc:686] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 LEADER]: Becoming Leader. State: Replica: 41ef6db1b6fa42c7b3078ef12adb4b18, State: Running, Role: LEADER
I20240502 14:00:03.948875 26258 consensus_queue.cc:237] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1020, Committed index: 1020, Last appended: 1.1020, Last appended by leader: 1020, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:03.951511 25506 catalog_manager.cc:5484] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 reported cstate change: term changed from 1 to 2, leader changed from a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67) to 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66). New cstate: current_term: 2 leader_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:03.982107 25654 raft_consensus.cc:1261] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Refusing update from remote peer 41ef6db1b6fa42c7b3078ef12adb4b18: Log matching property violated. Preceding OpId in replica: term: 1 index: 1020. Preceding OpId from leader: term: 2 index: 1022. (index mismatch)
I20240502 14:00:03.982669 26263 consensus_queue.cc:1035] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1021, Last known committed idx: 1020, Time since last communication: 0.000s
W20240502 14:00:03.982836 25718 consensus_peers.cc:485] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P 41ef6db1b6fa42c7b3078ef12adb4b18 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:03.990703 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.012341 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.037535 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.060968 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.138619 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.161901 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.189389 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.212895 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
I20240502 14:00:04.267414 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:04.294924 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.315644 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:04.333456 26004 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:04.348652 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.371809 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.461560 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.483906 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.518280 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.540855 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.635989 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.661530 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.695842 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.719518 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.823486 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.846016 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.889461 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.907891 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:04.957620 25965 meta_cache.cc:302] tablet a4e8027e5b0246f79e416a16331707ed: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:05.023206 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.040457 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.089716 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.107107 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.229427 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.240831 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.297477 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.311307 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.374312 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:05.437206 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.453857 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:05.488328 26209 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.488488 26209 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:05.488988 26209 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:05.489571 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 3 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" is_pre_election: true
I20240502 14:00:05.489753 25785 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 2.
I20240502 14:00:05.490149 25590 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters:
I20240502 14:00:05.490353 26209 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:00:05.490411 26209 raft_consensus.cc:482] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.490463 26209 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:00:05.490574 25587 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:05.491513 26209 raft_consensus.cc:504] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:05.491906 26209 leader_election.cc:288] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 election: Requested vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
I20240502 14:00:05.492309 25785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8027e5b0246f79e416a16331707ed" candidate_uuid: "f4eb887a0f284e20a3873c688d6cb667" candidate_term: 3 candidate_status { last_received { term: 1 index: 1020 } } ignore_live_leader: false dest_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18"
I20240502 14:00:05.492470 25785 raft_consensus.cc:3037] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:00:05.493218 25587 leader_election.cc:334] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:05.493574 25785 raft_consensus.cc:2454] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4eb887a0f284e20a3873c688d6cb667 in term 3.
I20240502 14:00:05.493921 25590 leader_election.cc:302] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41ef6db1b6fa42c7b3078ef12adb4b18, f4eb887a0f284e20a3873c688d6cb667; no voters: a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:05.494168 26209 raft_consensus.cc:2781] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:00:05.494395 26209 raft_consensus.cc:686] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [term 3 LEADER]: Becoming Leader. State: Replica: f4eb887a0f284e20a3873c688d6cb667, State: Running, Role: LEADER
I20240502 14:00:05.494573 26209 consensus_queue.cc:237] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1020, Committed index: 1020, Last appended: 1.1020, Last appended by leader: 1020, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:05.496660 25506 catalog_manager.cc:5484] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 reported cstate change: term changed from 1 to 3, leader changed from a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67) to f4eb887a0f284e20a3873c688d6cb667 (127.24.223.65). New cstate: current_term: 3 leader_uuid: "f4eb887a0f284e20a3873c688d6cb667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:05.510299 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
W20240502 14:00:05.527667 25745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48854: Illegal state: replica 41ef6db1b6fa42c7b3078ef12adb4b18 is not leader of this config: current role FOLLOWER
I20240502 14:00:05.586683 25785 raft_consensus.cc:1261] T a4e8027e5b0246f79e416a16331707ed P 41ef6db1b6fa42c7b3078ef12adb4b18 [term 3 FOLLOWER]: Refusing update from remote peer f4eb887a0f284e20a3873c688d6cb667: Log matching property violated. Preceding OpId in replica: term: 1 index: 1020. Preceding OpId from leader: term: 3 index: 1022. (index mismatch)
I20240502 14:00:05.587244 26209 consensus_queue.cc:1035] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1021, Last known committed idx: 1020, Time since last communication: 0.000s
W20240502 14:00:05.587513 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:05.768221 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 14:00:05.858251 26004 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f8a3f41fdcd google::LogMessage::Fail() at ??:0
@ 0x7f8a3f423b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8a3f41f7cc google::LogMessage::Flush() at ??:0
@ 0x7f8a3f420f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8a408015b1 kudu::operator-() at ??:0
@ 0x7f8a428e4fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f8a428ce05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f8a428cdf90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f8a428a49bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f8a434d0861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f8a434cf93a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7f8a434d8149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7f8a3e39ad09 std::function<>::operator()() at ??:0
@ 0x7f8a3e395107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f8a3e394a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f8a3e394123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f8a3e3bcb89 handle_request at ??:0
@ 0x7f8a3e3bf26b worker_thread at ??:0
@ 0x7f8a4109e6db start_thread at ??:0
@ 0x7f8a3e5d671f clone at ??:0
W20240502 14:00:06.090917 25590 connection.cc:535] client connection to 127.24.223.66:36681 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:06.090921 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
W20240502 14:00:06.092033 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:06.097702 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.115665 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.148757 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.189437 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.238065 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.263978 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:06.292528 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.358148 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.430156 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.511598 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.596941 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.603005 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.617728 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.647794 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.682426 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.708006 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.726764 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.786665 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.804395 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.871469 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.923713 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:06.941241 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.003402 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:07.020511 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:07.069221 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.093714 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.116667 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:07.162061 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.230326 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.306722 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.337468 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:07.351404 26336 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:07.351596 26336 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:07.352077 26336 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:07.353813 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:07.353940 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:07.354041 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:07.354241 26336 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:07.415665 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.501544 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.511157 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.621809 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.682448 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.776648 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.850589 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.929430 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:07.998024 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:00:07.998778 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:00:08.016793 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:08.037503 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.092140 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.099601 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:08.116582 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:08.149048 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:08.203116 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.297587 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.378206 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.502192 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.561846 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.693073 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.746479 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:08.787181 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:08.892589 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:08.964888 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:09.010915 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.100849 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.167027 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:09.206873 26336 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:09.207090 26336 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:09.207571 26336 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:09.208943 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:09.209043 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:09.209116 25587 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:09.209275 26336 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:09.247932 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.318799 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.471554 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.575320 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.701463 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.830507 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:09.944675 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:09.996776 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:10.095789 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.184904 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:10.195874 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.369304 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.412477 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:10.457790 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.633848 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.749341 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:10.762087 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:10.868247 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:10.907876 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:10.952440 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:11.040494 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:11.186008 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:11.240818 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:11.333729 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:11.342029 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:11.482609 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:11.657491 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:11.783316 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:11.941840 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:11.974546 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:12.094113 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:12.123098 26336 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:12.123250 26336 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:12.123732 26336 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:12.125016 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:12.125167 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:12.125253 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:12.125428 26336 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:12.302551 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:12.308578 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:12.415793 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:12.635890 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:12.761163 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:12.919149 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:12.973621 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:12.986200 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:13.112282 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:13.314983 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:13.324380 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 14:00:13.325134 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:13.351011 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:13.470875 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:13.471832 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:13.663769 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:13.846573 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:13.956281 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:14.014362 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:14.024088 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:14.225637 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:14.351824 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:14.393898 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:14.611979 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:14.774317 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:14.950490 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:15.014727 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:15.164294 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:15.419476 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:15.424002 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:15.493758 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:15.558780 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:15.801367 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:15.814370 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:15.828172 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:15.832592 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:15.897600 26336 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:15.897799 26336 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:15.898418 26336 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:15.900135 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:15.900354 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:15.900466 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:15.900681 26336 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:00:15.965921 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:16.115056 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:16.382136 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:16.523192 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:16.548596 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:16.717873 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:16.806448 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:16.958057 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:16.985061 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:17.237257 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:17.434268 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:17.529428 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:17.675920 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:17.843048 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:17.895085 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:17.965153 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:18.128938 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:18.326710 25590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 14:00:18.327711 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:18.328178 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:18.362259 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:18.591645 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:18.681516 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:18.838419 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:18.950181 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:19.058570 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:19.196727 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:19.320749 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:19.538568 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:19.702039 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:19.817536 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:19.957433 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:20.028216 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:20.318321 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:20.404386 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:20.527751 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:20.617928 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:20.664649 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:20.829620 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:20.862716 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:20.956547 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:21.033622 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:21.349624 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:21.545980 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:21.812855 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:21.876763 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:21.890869 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:21.957186 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:22.064126 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:22.412154 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:22.597836 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:22.859978 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:22.898993 26345 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:22.899192 26345 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:22.899706 26345 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:22.901051 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:22.901330 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:22.901404 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:22.901532 26345 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:22.902019 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:22.957199 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:22.962240 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:22.977916 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:23.137344 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:23.287181 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:23.336460 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:23.451653 25590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 14:00:23.689551 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:23.895397 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:23.928035 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:23.952167 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:24.247210 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:24.275122 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:24.464251 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:24.815060 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:24.994282 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:25.046116 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:25.397584 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:25.439640 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:25.507967 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:25.557354 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:25.638661 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:25.738835 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:25.987571 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:25.996320 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:26.237155 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:26.277536 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:26.587244 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:26.642542 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:26.843890 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:26.967483 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:00:27.036623 26345 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:27.036803 26345 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:27.037398 26345 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:27.038755 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:27.038870 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:27.038934 25587 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:27.039130 26345 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:27.197162 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:27.456867 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:27.663637 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:27.812505 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:27.836196 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:00:27.958032 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:27.966439 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:28.081430 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:28.109918 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:28.433156 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:28.504657 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 14:00:28.718461 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:28.793988 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:28.853773 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:28.961588 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:29.066088 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:29.357734 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:29.705598 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:29.921272 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:29.958318 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:00:29.982882 26345 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:29.983075 26345 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:29.983603 26345 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:29.984844 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:29.985157 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:29.985266 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:29.985420 26345 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:30.009285 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:30.016520 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:30.354867 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:30.520507 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:30.680734 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:30.693760 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:00:31.002264 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:31.016928 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:31.126304 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:31.349701 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:31.607304 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:31.684531 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:31.974714 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:32.023828 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:32.133242 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:32.358191 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:32.564723 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:32.714349 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:32.891557 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:33.044152 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:33.051713 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:33.182677 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:33.418093 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:33.449894 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:33.523949 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:00:33.737840 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:34.014693 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:00:34.049167 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:34.127784 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:34.207868 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:34.444427 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:34.843348 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:35.018436 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:00:35.057135 26345 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:35.057315 26345 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:35.057775 26345 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:35.058996 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:35.059170 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:35.059257 25587 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:35.059415 26345 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:35.160487 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:35.293119 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:35.326762 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:35.571430 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:35.616787 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:35.887280 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:35.973181 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:36.078083 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:36.306748 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:36.372673 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:36.616340 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:36.647655 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:00:36.975703 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:37.046412 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:37.361472 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:37.547058 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:37.799939 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:37.920475 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:00:38.010628 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:38.033850 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:38.113214 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:38.427348 26345 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:38.427537 26345 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:38.428104 26345 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:38.429397 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:38.429896 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:38.429970 25587 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:38.430178 26345 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:38.547492 25590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:00:38.548353 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:38.556699 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:38.618162 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:38.871949 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:39.009677 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:39.184424 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:39.331338 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:39.638345 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:39.849902 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:40.031603 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:40.108672 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:40.411340 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:40.498239 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:40.734371 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:40.899259 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:40.936208 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:41.018069 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:41.147140 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:41.200201 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:41.546170 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:41.691516 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:41.957746 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:41.993839 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:42.001369 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:42.494817 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:42.672552 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:42.802184 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:42.997516 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:43.030896 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:43.035158 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:43.306546 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:43.607594 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:00:43.615638 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:43.704556 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:00:43.962724 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:44.116992 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:44.128031 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:44.161440 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:44.437659 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:44.958894 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:44.968084 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:44.986531 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:00:45.239209 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:45.267577 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:45.524776 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:45.803256 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:45.989534 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:46.102406 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:46.372804 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:46.382647 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:46.484776 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:46.656095 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:46.945091 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:46.968140 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:47.475582 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:00:47.516582 25612 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:47.516620 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:47.801235 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:47.956796 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:47.982640 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:00:48.096249 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:48.664167 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:48.689154 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:00:48.895742 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:00:48.955734 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:48.970856 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:48.986490 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:49.072388 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:00:49.539700 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:49.856958 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:49.966539 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:50.127029 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:50.150763 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:00:50.209570 26357 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:50.209728 26357 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:50.210228 26357 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:50.211519 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:50.211616 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
I20240502 14:00:50.211707 25587 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:50.211870 26357 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:50.424247 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:50.697837 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:00:50.748808 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:50.967720 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:51.318778 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:51.352308 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:51.366866 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:00:51.598522 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:00:51.654670 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:51.957396 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:52.218820 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:52.520557 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:52.564705 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:52.624779 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:00:53.064961 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:53.129772 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:53.410641 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:00:53.483368 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:53.715055 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:00:53.742621 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:53.956876 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:54.049814 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:54.054116 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:00:54.112998 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:00:54.410431 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:54.965564 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:54.979018 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:55.012794 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:00:55.033057 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:55.343829 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:55.914813 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:55.958201 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:56.142112 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:00:56.230027 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:56.295140 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:56.555353 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:00:56.607025 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:00:56.866357 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:56.980676 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:57.250408 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:57.292646 26357 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:00:57.292824 26357 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:00:57.293316 26357 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:00:57.294605 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:57.294790 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:57.294852 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:00:57.295059 26357 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:57.336644 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:00:57.504158 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:00:57.821727 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:00:57.986558 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:58.211411 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:58.454097 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:00:58.532472 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:00:58.787668 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:58.832306 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:00:58.978683 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:00:59.168121 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:00:59.182978 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:59.278630 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:00:59.671234 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:00:59.766781 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:00:59.841198 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:00:59.981878 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:01:00.167106 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:00.746743 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:00.869851 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:01:00.994503 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:01:01.075462 25965 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:01:01.159644 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:01.740137 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:01.752218 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:01.754758 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:01:01.976681 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:01:02.159266 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:02.411653 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:01:02.445672 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:01:02.502918 26361 raft_consensus.cc:482] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 41ef6db1b6fa42c7b3078ef12adb4b18)
I20240502 14:01:02.503166 26361 raft_consensus.cc:504] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb887a0f284e20a3873c688d6cb667" member_type: VOTER last_known_addr { host: "127.24.223.65" port: 37497 } } peers { permanent_uuid: "41ef6db1b6fa42c7b3078ef12adb4b18" member_type: VOTER last_known_addr { host: "127.24.223.66" port: 36681 } } peers { permanent_uuid: "a0e1f3276aa441ffba0ab2cdb7e7cbdf" member_type: VOTER last_known_addr { host: "127.24.223.67" port: 39315 } }
I20240502 14:01:02.503816 26361 leader_election.cc:288] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681), a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315)
W20240502 14:01:02.505224 25587 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111)
W20240502 14:01:02.505365 25590 leader_election.cc:334] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
I20240502 14:01:02.505421 25590 leader_election.cc:302] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4eb887a0f284e20a3873c688d6cb667; no voters: 41ef6db1b6fa42c7b3078ef12adb4b18, a0e1f3276aa441ffba0ab2cdb7e7cbdf
I20240502 14:01:02.505590 26361 raft_consensus.cc:2726] T 4ffa3ff04e7e48bea816ec91e7e1bd10 P f4eb887a0f284e20a3873c688d6cb667 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:02.747637 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:01:03.004272 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:01:03.005681 26005 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/26360/stat: No such file or directory (error 2)
W20240502 14:01:03.167536 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:03.406831 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:03.419312 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:01:03.758644 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:03.900439 25587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:01:03.970886 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:01:04.152747 25590 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:04.164983 25590 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681): Couldn't send request to peer 41ef6db1b6fa42c7b3078ef12adb4b18. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:04.191434 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:04.437808 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:01:04.780371 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
I20240502 14:01:04.973623 26005 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:01:05.067282 25587 consensus_peers.cc:485] T 3fdd9917be1545c996d33376fee706ce P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:05.217181 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:05.463972 25968 meta_cache.cc:302] tablet 4ffa3ff04e7e48bea816ec91e7e1bd10: replica 41ef6db1b6fa42c7b3078ef12adb4b18 (127.24.223.66:36681) has failed: Network error: Client connection negotiation failed: client connection to 127.24.223.66:36681: connect: Connection refused (error 111)
W20240502 14:01:05.805636 25614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57942: Illegal state: replica f4eb887a0f284e20a3873c688d6cb667 is not leader of this config: current role FOLLOWER
W20240502 14:01:05.907224 25965 batcher.cc:441] Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907634 25469 client-test-util.cc:49] 9 failed ops. First 10 errors follow
I20240502 14:01:05.907712 25469 client-test-util.cc:55] Op INSERT int64 rand_key=6353742114144518147, int64 link_to=-3744064838872334333, int64 insert_ts=1714658405855478 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907866 25469 client-test-util.cc:55] Op INSERT int64 rand_key=7369983649220591623, int64 link_to=7912544385618018311, int64 insert_ts=1714658405855576 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907895 25469 client-test-util.cc:55] Op INSERT int64 rand_key=8385802971831664651, int64 link_to=1122550273887109131, int64 insert_ts=1714658405855670 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907933 25469 client-test-util.cc:55] Op INSERT int64 rand_key=6968505857845231628, int64 link_to=3470062505755344908, int64 insert_ts=1714658405855693 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907964 25469 client-test-util.cc:55] Op INSERT int64 rand_key=7984606655433015312, int64 link_to=-3319931605975564272, int64 insert_ts=1714658405855782 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.907994 25469 client-test-util.cc:55] Op INSERT int64 rand_key=9000566715532378132, int64 link_to=8336818356003078164, int64 insert_ts=1714658405855879 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.908027 25469 client-test-util.cc:55] Op INSERT int64 rand_key=6710946316762284065, int64 link_to=3367650821661327393, int64 insert_ts=1714658405856195 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.908066 25469 client-test-util.cc:55] Op INSERT int64 rand_key=7726906376861646885, int64 link_to=-3422343290069581787, int64 insert_ts=1714658405856282 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
I20240502 14:01:05.908104 25469 client-test-util.cc:55] Op INSERT int64 rand_key=8743007174449430569, int64 link_to=8234406671909060649, int64 insert_ts=1714658405856369 had status Timed out: Failed to write batch of 9 ops to tablet 4ffa3ff04e7e48bea816ec91e7e1bd10 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 4ffa3ff04e7e48bea816ec91e7e1bd10, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.24.223.66:36681 (error 108)
F20240502 14:01:05.908154 25469 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714658465 (unix time) try "date -d @1714658465" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000637d) received by PID 25469 (TID 0x7f342e64b280) from PID 25469; stack trace: ***
@ 0x7f343a962980 (unknown) at ??:0
@ 0x7f34385e6fb7 gsignal at ??:0
@ 0x7f34385e8921 abort at ??:0
@ 0x7f34394f3dcd google::LogMessage::Fail() at ??:0
@ 0x7f34394f7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f34394f37cc google::LogMessage::Flush() at ??:0
@ 0x7f34394f4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f343bc257f2 kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x5567982d4236 kudu::client::FlushSessionOrDie() at ??:0
@ 0x5567982d7caa kudu::LinkedListTester::LoadLinkedList() at ??:0
@ 0x5567982cd4ca kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at ??:0
@ 0x7f343a3ea0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f343a3debdb testing::Test::Run() at ??:0
@ 0x7f343a3ded9d testing::TestInfo::Run() at ??:0
@ 0x7f343a3df377 testing::TestSuite::Run() at ??:0
@ 0x7f343a3df77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f343a3ea60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f343a3dee63 testing::UnitTest::Run() at ??:0
@ 0x7f343bc371de RUN_ALL_TESTS() at ??:0
@ 0x7f343bc3653d main at ??:0
@ 0x7f34385c9bf7 __libc_start_main at ??:0
@ 0x5567982cb34a _start at ??:0
W20240502 14:01:06.016741 25587 consensus_peers.cc:485] T a4e8027e5b0246f79e416a16331707ed P f4eb887a0f284e20a3873c688d6cb667 -> Peer a0e1f3276aa441ffba0ab2cdb7e7cbdf (127.24.223.67:39315): Couldn't send request to peer a0e1f3276aa441ffba0ab2cdb7e7cbdf. Status: Network error: Client connection negotiation failed: client connection to 127.24.223.67:39315: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
E20240502 14:01:06.136197 26038 webserver.cc:533] Webserver: error reading: Connection reset by peer