[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:57:57.505379 1050 test_util.cc:268] Using random seed: -291567277
I20240502 01:57:57.511785 1050 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 01:57:57.511844 1050 linked_list-test.cc:101] --------------
I20240502 01:57:57.511858 1050 linked_list-test.cc:102] 50 chains
I20240502 01:57:57.511875 1050 linked_list-test.cc:103] 3 tablets
I20240502 01:57:57.511895 1050 linked_list-test.cc:104] Mutations on
I20240502 01:57:57.511911 1050 linked_list-test.cc:105] --------------
I20240502 01:57:57.511994 1050 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:57:57.512022 1050 ts_itest-base.cc:113] --------------
I20240502 01:57:57.512034 1050 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:57:57.512048 1050 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:57:57.512059 1050 ts_itest-base.cc:116] --------------
2024-05-02T01:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:57:57Z Disabled control of system clock
I20240502 01:57:57.521692 1050 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
/tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.6.190:35639
--webserver_interface=127.1.6.190
--webserver_port=0
--builtin_ntp_servers=127.1.6.148:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.6.190:35639
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:57:57.623136 1062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:57.623342 1062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:57.623389 1062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:57.626243 1062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:57:57.626291 1062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:57.626309 1062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:57:57.626327 1062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:57:57.629667 1062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.6.148:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.6.190:35639
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.6.190:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.1.6.190
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.1062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:57.630471 1062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:57.631492 1062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:57.639273 1067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.639897 1070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.640560 1068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:57.641440 1062 server_base.cc:1034] running on GCE node
I20240502 01:57:57.642091 1062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:57.642992 1062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:57.644155 1062 hybrid_clock.cc:648] HybridClock initialized: now 1714615077644135 us; error 34 us; skew 500 ppm
I20240502 01:57:57.649849 1062 webserver.cc:460] Webserver started at http://127.1.6.190:44199/ using document root <none> and password file <none>
I20240502 01:57:57.650425 1062 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:57.650481 1062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:57.650657 1062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:57.652271 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data/instance:
uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.652740 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal/instance:
uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.655921 1062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:57.657992 1077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.658982 1062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:57.659101 1062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal
uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.659181 1062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:57.675642 1062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:57.676261 1062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:57.676412 1062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:57.682837 1062 rpc_server.cc:307] RPC server started. Bound to: 127.1.6.190:35639
I20240502 01:57:57.682852 1130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.6.190:35639 every 8 connection(s)
I20240502 01:57:57.683753 1062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/data/info.pb
I20240502 01:57:57.686203 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:57.686666 1050 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu as pid 1062
I20240502 01:57:57.686786 1050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/master-0/wal/instance
I20240502 01:57:57.691177 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5: Bootstrap starting.
I20240502 01:57:57.693228 1131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:57.694022 1131 log.cc:824] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:57.695709 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5: No bootstrap required, opened a new log
I20240502 01:57:57.698626 1131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } }
I20240502 01:57:57.698827 1131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:57.698923 1131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3b6bf1b310475fb2f22a4d256e6fe5, State: Initialized, Role: FOLLOWER
I20240502 01:57:57.699371 1131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } }
I20240502 01:57:57.699517 1131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:57.699596 1131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:57.699705 1131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:57.700485 1131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } }
I20240502 01:57:57.700812 1131 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3b6bf1b310475fb2f22a4d256e6fe5; no voters:
I20240502 01:57:57.701083 1131 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:57.701454 1136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:57.701665 1136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3b6bf1b310475fb2f22a4d256e6fe5, State: Running, Role: LEADER
I20240502 01:57:57.701920 1136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } }
I20240502 01:57:57.702178 1131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:57.703342 1138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3b6bf1b310475fb2f22a4d256e6fe5. Latest consensus state: current_term: 1 leader_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } } }
I20240502 01:57:57.703465 1138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:57.703768 1137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3b6bf1b310475fb2f22a4d256e6fe5" member_type: VOTER last_known_addr { host: "127.1.6.190" port: 35639 } } }
I20240502 01:57:57.703815 1145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:57.703871 1137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:57.706519 1145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:57.711455 1145 catalog_manager.cc:1347] Generated new cluster ID: 61f688676f44421a9be23f8370e784bb
I20240502 01:57:57.711509 1145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:57.722116 1145 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:57.723168 1145 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:57.736056 1145 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2d3b6bf1b310475fb2f22a4d256e6fe5: Generated new TSK 0
I20240502 01:57:57.736888 1145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:57.743206 1050 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
/tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.6.129:0
--local_ip_for_outbound_sockets=127.1.6.129
--webserver_interface=127.1.6.129
--webserver_port=0
--tserver_master_addrs=127.1.6.190:35639
--builtin_ntp_servers=127.1.6.148:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 01:57:57.852968 1155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:57.853192 1155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:57.853272 1155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:57.856280 1155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:57.856357 1155 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:57:57.856377 1155 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:57:57.856407 1155 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:57:57.856492 1155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.6.129
I20240502 01:57:57.860069 1155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.6.148:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.1.6.129
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.6.190:35639
--never_fsync=true
--heap_profile_path=/tmp/kudu.1155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.6.129
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:57.860944 1155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:57.862069 1155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:57.870455 1163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.871028 1161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.871546 1160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:57.872368 1155 server_base.cc:1034] running on GCE node
I20240502 01:57:57.872715 1155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:57.873234 1155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:57.874382 1155 hybrid_clock.cc:648] HybridClock initialized: now 1714615077874369 us; error 38 us; skew 500 ppm
I20240502 01:57:57.876104 1155 webserver.cc:460] Webserver started at http://127.1.6.129:46371/ using document root <none> and password file <none>
I20240502 01:57:57.876657 1155 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:57.876736 1155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:57.876910 1155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:57.878542 1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data/instance:
uuid: "af730d19ec604080a589a5bd48fb7786"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.878980 1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal/instance:
uuid: "af730d19ec604080a589a5bd48fb7786"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.882122 1155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:57.884142 1169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.885035 1155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:57.885154 1155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal
uuid: "af730d19ec604080a589a5bd48fb7786"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-6xpb"
I20240502 01:57:57.885232 1155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:57.894891 1155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:57.895406 1155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:57.895551 1155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:57.896011 1155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:57.896806 1155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:57.896854 1155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.896906 1155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:57.896940 1155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.905128 1155 rpc_server.cc:307] RPC server started. Bound to: 127.1.6.129:39239
I20240502 01:57:57.905169 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.6.129:39239 every 8 connection(s)
I20240502 01:57:57.905966 1155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:57:57.911060 1050 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu as pid 1155
I20240502 01:57:57.911159 1050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-0/wal/instance
I20240502 01:57:57.913208 1050 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
/tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.6.130:0
--local_ip_for_outbound_sockets=127.1.6.130
--webserver_interface=127.1.6.130
--webserver_port=0
--tserver_master_addrs=127.1.6.190:35639
--builtin_ntp_servers=127.1.6.148:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 01:57:57.914398 1284 heartbeater.cc:347] Connected to a master server at 127.1.6.190:35639
I20240502 01:57:57.914665 1284 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:57.915302 1284 heartbeater.cc:511] Master 127.1.6.190:35639 requested a full tablet report, sending...
I20240502 01:57:57.916618 1094 ts_manager.cc:194] Registered new tserver with Master: af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239)
I20240502 01:57:57.917744 1094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.6.129:45075
W20240502 01:57:58.016203 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:58.016422 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:58.016507 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:58.019572 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:58.019654 1287 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:57:58.019688 1287 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:57:58.019737 1287 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:57:58.019817 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.6.130
I20240502 01:57:58.023257 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.6.148:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.1.6.130
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.6.190:35639
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.6.130
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:58.024066 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:58.025209 1287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.032986 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.033468 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.034206 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.034876 1287 server_base.cc:1034] running on GCE node
I20240502 01:57:58.035188 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:58.035646 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:58.036798 1287 hybrid_clock.cc:648] HybridClock initialized: now 1714615078036773 us; error 36 us; skew 500 ppm
I20240502 01:57:58.038647 1287 webserver.cc:460] Webserver started at http://127.1.6.130:45833/ using document root <none> and password file <none>
I20240502 01:57:58.039130 1287 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.039181 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.039340 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.040822 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data/instance:
uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.041249 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal/instance:
uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.044349 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:58.046306 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.047246 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.047336 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal
uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.047410 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.071484 1287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.072057 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:58.072180 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.072638 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:58.073486 1287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:58.073534 1287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.073573 1287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:58.073596 1287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.082005 1287 rpc_server.cc:307] RPC server started. Bound to: 127.1.6.130:34713
I20240502 01:57:58.082024 1416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.6.130:34713 every 8 connection(s)
I20240502 01:57:58.082917 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/data/info.pb
I20240502 01:57:58.088920 1050 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu as pid 1287
I20240502 01:57:58.089074 1050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-1/wal/instance
I20240502 01:57:58.091146 1050 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
/tmp/dist-test-taskQEsr5p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.6.131:0
--local_ip_for_outbound_sockets=127.1.6.131
--webserver_interface=127.1.6.131
--webserver_port=0
--tserver_master_addrs=127.1.6.190:35639
--builtin_ntp_servers=127.1.6.148:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 01:57:58.091249 1417 heartbeater.cc:347] Connected to a master server at 127.1.6.190:35639
I20240502 01:57:58.091444 1417 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:58.092001 1417 heartbeater.cc:511] Master 127.1.6.190:35639 requested a full tablet report, sending...
I20240502 01:57:58.093075 1094 ts_manager.cc:194] Registered new tserver with Master: 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
I20240502 01:57:58.093611 1094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.6.130:57903
W20240502 01:57:58.192771 1420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:58.192974 1420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:58.193058 1420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:58.195879 1420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:58.195955 1420 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:57:58.195993 1420 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:57:58.196028 1420 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:57:58.196099 1420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.6.131
I20240502 01:57:58.199443 1420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.6.148:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.1.6.131
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.6.190:35639
--never_fsync=true
--heap_profile_path=/tmp/kudu.1420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.6.131
--log_dir=/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:58.200204 1420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:58.201251 1420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.209342 1426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.209841 1425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.210631 1428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.211179 1420 server_base.cc:1034] running on GCE node
I20240502 01:57:58.211514 1420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:58.212003 1420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:58.213151 1420 hybrid_clock.cc:648] HybridClock initialized: now 1714615078213143 us; error 23 us; skew 500 ppm
I20240502 01:57:58.214946 1420 webserver.cc:460] Webserver started at http://127.1.6.131:40365/ using document root <none> and password file <none>
I20240502 01:57:58.215457 1420 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.215515 1420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.215678 1420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.217286 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data/instance:
uuid: "778cc9c9c52648c28e0f04ccf879d34b"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.217746 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal/instance:
uuid: "778cc9c9c52648c28e0f04ccf879d34b"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.220857 1420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:58.222901 1434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.223819 1420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.223924 1420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal
uuid: "778cc9c9c52648c28e0f04ccf879d34b"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-6xpb"
I20240502 01:57:58.224004 1420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.240280 1420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.240839 1420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:58.240995 1420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.241484 1420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:58.242380 1420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:58.242430 1420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.242484 1420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:58.242519 1420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.250797 1420 rpc_server.cc:307] RPC server started. Bound to: 127.1.6.131:34361
I20240502 01:57:58.250828 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.6.131:34361 every 8 connection(s)
I20240502 01:57:58.251675 1420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/data/info.pb
I20240502 01:57:58.256496 1050 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQEsr5p/build/debug/bin/kudu as pid 1420
I20240502 01:57:58.256615 1050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQEsr5p/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714615077495333-1050-0/linked-list-cluster/ts-2/wal/instance
I20240502 01:57:58.259940 1550 heartbeater.cc:347] Connected to a master server at 127.1.6.190:35639
I20240502 01:57:58.260123 1550 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:58.260537 1550 heartbeater.cc:511] Master 127.1.6.190:35639 requested a full tablet report, sending...
I20240502 01:57:58.261344 1094 ts_manager.cc:194] Registered new tserver with Master: 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
I20240502 01:57:58.261835 1094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.6.131:52999
I20240502 01:57:58.270920 1050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:57:58.281944 1094 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53434:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 01:57:58.284678 1094 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:57:58.300313 1483 tablet_service.cc:1467] Processing CreateTablet for tablet a293fda3c5f24da2a05338b3bbcebe70 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:57:58.300897 1348 tablet_service.cc:1467] Processing CreateTablet for tablet 37eeb952913c42f7b0118e2c508bdc46 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:57:58.301539 1483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a293fda3c5f24da2a05338b3bbcebe70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.301671 1347 tablet_service.cc:1467] Processing CreateTablet for tablet e54c9f86c1b84a6f8c0a773e029392d3 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:57:58.301261 1349 tablet_service.cc:1467] Processing CreateTablet for tablet a293fda3c5f24da2a05338b3bbcebe70 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:57:58.301993 1348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eeb952913c42f7b0118e2c508bdc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.300313 1481 tablet_service.cc:1467] Processing CreateTablet for tablet e54c9f86c1b84a6f8c0a773e029392d3 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:57:58.302911 1481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54c9f86c1b84a6f8c0a773e029392d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.302448 1215 tablet_service.cc:1467] Processing CreateTablet for tablet e54c9f86c1b84a6f8c0a773e029392d3 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:57:58.302459 1217 tablet_service.cc:1467] Processing CreateTablet for tablet a293fda3c5f24da2a05338b3bbcebe70 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:57:58.303520 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54c9f86c1b84a6f8c0a773e029392d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.303886 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a293fda3c5f24da2a05338b3bbcebe70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.306296 1216 tablet_service.cc:1467] Processing CreateTablet for tablet 37eeb952913c42f7b0118e2c508bdc46 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:57:58.306560 1216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eeb952913c42f7b0118e2c508bdc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.307039 1347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54c9f86c1b84a6f8c0a773e029392d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.309396 1482 tablet_service.cc:1467] Processing CreateTablet for tablet 37eeb952913c42f7b0118e2c508bdc46 (DEFAULT_TABLE table=TestTable [id=f09e6244f6c04f2b9ba1639f8c741951]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:57:58.309715 1482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eeb952913c42f7b0118e2c508bdc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.309996 1568 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Bootstrap starting.
I20240502 01:57:58.311918 1349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a293fda3c5f24da2a05338b3bbcebe70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.312256 1568 tablet_bootstrap.cc:654] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.313162 1569 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: Bootstrap starting.
I20240502 01:57:58.313411 1568 log.cc:824] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:58.314204 1567 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Bootstrap starting.
I20240502 01:57:58.315028 1569 tablet_bootstrap.cc:654] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.315259 1568 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: No bootstrap required, opened a new log
I20240502 01:57:58.315429 1568 ts_tablet_manager.cc:1367] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:57:58.315941 1569 log.cc:824] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:58.316095 1567 tablet_bootstrap.cc:654] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.317119 1567 log.cc:824] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:58.317471 1569 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: No bootstrap required, opened a new log
I20240502 01:57:58.317643 1569 ts_tablet_manager.cc:1367] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:57:58.318696 1567 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: No bootstrap required, opened a new log
I20240502 01:57:58.318879 1567 ts_tablet_manager.cc:1367] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:57:58.318779 1568 raft_consensus.cc:348] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.318987 1568 raft_consensus.cc:374] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.319046 1568 raft_consensus.cc:729] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af730d19ec604080a589a5bd48fb7786, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.319494 1568 consensus_queue.cc:260] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.320163 1284 heartbeater.cc:503] Master 127.1.6.190:35639 was elected leader, sending a full tablet report...
I20240502 01:57:58.320317 1568 ts_tablet_manager.cc:1398] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240502 01:57:58.320542 1568 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786: Bootstrap starting.
I20240502 01:57:58.320537 1569 raft_consensus.cc:348] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.320708 1569 raft_consensus.cc:374] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.320760 1569 raft_consensus.cc:729] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778cc9c9c52648c28e0f04ccf879d34b, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.321148 1569 consensus_queue.cc:260] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.321434 1568 tablet_bootstrap.cc:654] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.321894 1569 ts_tablet_manager.cc:1398] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:57:58.321827 1567 raft_consensus.cc:348] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.322007 1567 raft_consensus.cc:374] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.322103 1567 raft_consensus.cc:729] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e0a5214cbc4a15b6b8aacd66ccb60c, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.322238 1550 heartbeater.cc:503] Master 127.1.6.190:35639 was elected leader, sending a full tablet report...
I20240502 01:57:58.322523 1568 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786: No bootstrap required, opened a new log
I20240502 01:57:58.322599 1568 ts_tablet_manager.cc:1367] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:58.322819 1567 consensus_queue.cc:260] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.323104 1568 raft_consensus.cc:348] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.323191 1568 raft_consensus.cc:374] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.323220 1568 raft_consensus.cc:729] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af730d19ec604080a589a5bd48fb7786, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.323325 1568 consensus_queue.cc:260] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.323590 1567 ts_tablet_manager.cc:1398] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:57:58.323638 1568 ts_tablet_manager.cc:1398] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:57:58.323781 1568 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786: Bootstrap starting.
I20240502 01:57:58.323895 1417 heartbeater.cc:503] Master 127.1.6.190:35639 was elected leader, sending a full tablet report...
I20240502 01:57:58.324646 1568 tablet_bootstrap.cc:654] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.324882 1569 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: Bootstrap starting.
I20240502 01:57:58.325066 1567 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Bootstrap starting.
I20240502 01:57:58.325924 1569 tablet_bootstrap.cc:654] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.325968 1567 tablet_bootstrap.cc:654] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.326349 1568 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786: No bootstrap required, opened a new log
I20240502 01:57:58.326416 1568 ts_tablet_manager.cc:1367] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:57:58.326844 1568 raft_consensus.cc:348] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.326925 1568 raft_consensus.cc:374] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.326959 1568 raft_consensus.cc:729] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af730d19ec604080a589a5bd48fb7786, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.327064 1567 tablet_bootstrap.cc:492] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c: No bootstrap required, opened a new log
I20240502 01:57:58.327064 1568 consensus_queue.cc:260] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.327131 1567 ts_tablet_manager.cc:1367] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.327189 1569 tablet_bootstrap.cc:492] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: No bootstrap required, opened a new log
I20240502 01:57:58.327261 1569 ts_tablet_manager.cc:1367] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.327387 1568 ts_tablet_manager.cc:1398] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.327632 1567 raft_consensus.cc:348] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.327715 1569 raft_consensus.cc:348] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.327767 1567 raft_consensus.cc:374] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.327819 1567 raft_consensus.cc:729] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e0a5214cbc4a15b6b8aacd66ccb60c, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.327826 1569 raft_consensus.cc:374] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.327865 1569 raft_consensus.cc:729] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778cc9c9c52648c28e0f04ccf879d34b, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.327967 1567 consensus_queue.cc:260] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.328027 1569 consensus_queue.cc:260] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.328294 1567 ts_tablet_manager.cc:1398] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.328428 1569 ts_tablet_manager.cc:1398] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.328594 1569 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b: Bootstrap starting.
I20240502 01:57:58.329284 1567 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Bootstrap starting.
I20240502 01:57:58.329478 1569 tablet_bootstrap.cc:654] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.330761 1569 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b: No bootstrap required, opened a new log
I20240502 01:57:58.330854 1569 ts_tablet_manager.cc:1367] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:57:58.330873 1567 tablet_bootstrap.cc:654] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.331331 1569 raft_consensus.cc:348] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.331436 1569 raft_consensus.cc:374] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.331468 1569 raft_consensus.cc:729] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778cc9c9c52648c28e0f04ccf879d34b, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.331581 1569 consensus_queue.cc:260] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.331812 1569 ts_tablet_manager.cc:1398] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.331926 1567 tablet_bootstrap.cc:492] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c: No bootstrap required, opened a new log
I20240502 01:57:58.332023 1567 ts_tablet_manager.cc:1367] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:57:58.332442 1567 raft_consensus.cc:348] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.332542 1567 raft_consensus.cc:374] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.332576 1567 raft_consensus.cc:729] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e0a5214cbc4a15b6b8aacd66ccb60c, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.332680 1567 consensus_queue.cc:260] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.332947 1567 ts_tablet_manager.cc:1398] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:57:58.333786 1418 tablet.cc:2369] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:58.359118 1584 raft_consensus.cc:482] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:58.359331 1584 raft_consensus.cc:504] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.360502 1584 leader_election.cc:288] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
I20240502 01:57:58.364127 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" is_pre_election: true
I20240502 01:57:58.364449 1369 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 0.
I20240502 01:57:58.364612 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786" is_pre_election: true
I20240502 01:57:58.364859 1237 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 0.
I20240502 01:57:58.364885 1436 leader_election.cc:302] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, 778cc9c9c52648c28e0f04ccf879d34b; no voters:
I20240502 01:57:58.365157 1584 raft_consensus.cc:2781] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:58.365234 1584 raft_consensus.cc:482] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:58.365286 1584 raft_consensus.cc:3037] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.366179 1584 raft_consensus.cc:504] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.366465 1584 leader_election.cc:288] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 election: Requested vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
I20240502 01:57:58.366763 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786"
I20240502 01:57:58.366775 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
I20240502 01:57:58.366854 1237 raft_consensus.cc:3037] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.366863 1369 raft_consensus.cc:3037] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.367686 1237 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 1.
I20240502 01:57:58.367794 1369 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 1.
I20240502 01:57:58.367959 1438 leader_election.cc:302] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:57:58.368120 1584 raft_consensus.cc:2781] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.368211 1584 raft_consensus.cc:686] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 LEADER]: Becoming Leader. State: Replica: 778cc9c9c52648c28e0f04ccf879d34b, State: Running, Role: LEADER
I20240502 01:57:58.368518 1584 consensus_queue.cc:237] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:57:58.371390 1093 catalog_manager.cc:5484] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b reported cstate change: term changed from 0 to 1, leader changed from <none> to 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131). New cstate: current_term: 1 leader_uuid: "778cc9c9c52648c28e0f04ccf879d34b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } health_report { overall_health: UNKNOWN } } }
I20240502 01:57:58.406733 1583 raft_consensus.cc:482] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:58.406884 1583 raft_consensus.cc:504] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
W20240502 01:57:58.407075 1285 tablet.cc:2369] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:58.407980 1583 leader_election.cc:288] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
I20240502 01:57:58.411633 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54c9f86c1b84a6f8c0a773e029392d3" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786" is_pre_election: true
I20240502 01:57:58.411798 1237 raft_consensus.cc:2454] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 0.
I20240502 01:57:58.412015 1503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54c9f86c1b84a6f8c0a773e029392d3" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "778cc9c9c52648c28e0f04ccf879d34b" is_pre_election: true
I20240502 01:57:58.412169 1305 leader_election.cc:302] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:57:58.412225 1503 raft_consensus.cc:2454] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 0.
I20240502 01:57:58.412386 1583 raft_consensus.cc:2781] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:58.412449 1583 raft_consensus.cc:482] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:58.412497 1583 raft_consensus.cc:3037] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.413421 1583 raft_consensus.cc:504] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.413764 1583 leader_election.cc:288] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 1 election: Requested vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
I20240502 01:57:58.414090 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54c9f86c1b84a6f8c0a773e029392d3" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786"
I20240502 01:57:58.414193 1503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54c9f86c1b84a6f8c0a773e029392d3" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "778cc9c9c52648c28e0f04ccf879d34b"
I20240502 01:57:58.414256 1237 raft_consensus.cc:3037] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.414325 1503 raft_consensus.cc:3037] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.415113 1237 raft_consensus.cc:2454] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 1.
I20240502 01:57:58.415242 1503 raft_consensus.cc:2454] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 1.
I20240502 01:57:58.415417 1305 leader_election.cc:302] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:57:58.415627 1583 raft_consensus.cc:2781] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.415729 1583 raft_consensus.cc:686] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 LEADER]: Becoming Leader. State: Replica: 47e0a5214cbc4a15b6b8aacd66ccb60c, State: Running, Role: LEADER
I20240502 01:57:58.416008 1583 consensus_queue.cc:237] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.418958 1093 catalog_manager.cc:5484] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130). New cstate: current_term: 1 leader_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } health_report { overall_health: UNKNOWN } } }
I20240502 01:57:58.477588 1584 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:58.477723 1584 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.478253 1584 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
I20240502 01:57:58.478706 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786" is_pre_election: true
I20240502 01:57:58.478708 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" is_pre_election: true
I20240502 01:57:58.478832 1237 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 0.
I20240502 01:57:58.478855 1369 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 0.
I20240502 01:57:58.479120 1438 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:57:58.479275 1584 raft_consensus.cc:2781] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:58.479332 1584 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:58.479359 1584 raft_consensus.cc:3037] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.480162 1584 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.480530 1584 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 election: Requested vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
I20240502 01:57:58.480875 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786"
I20240502 01:57:58.480895 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "778cc9c9c52648c28e0f04ccf879d34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
I20240502 01:57:58.480996 1237 raft_consensus.cc:3037] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.481034 1369 raft_consensus.cc:3037] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.481767 1237 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 1.
I20240502 01:57:58.481796 1369 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778cc9c9c52648c28e0f04ccf879d34b in term 1.
I20240502 01:57:58.482015 1438 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:57:58.482182 1584 raft_consensus.cc:2781] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.482261 1584 raft_consensus.cc:686] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 LEADER]: Becoming Leader. State: Replica: 778cc9c9c52648c28e0f04ccf879d34b, State: Running, Role: LEADER
I20240502 01:57:58.482412 1584 consensus_queue.cc:237] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:57:58.483979 1093 catalog_manager.cc:5484] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b reported cstate change: term changed from 0 to 1, leader changed from <none> to 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131). New cstate: current_term: 1 leader_uuid: "778cc9c9c52648c28e0f04ccf879d34b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } health_report { overall_health: HEALTHY } } }
W20240502 01:57:58.502497 1551 tablet.cc:2369] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:58.529016 1050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:57:58.530725 1050 ts_itest-base.cc:233] Waiting for 3 tablets on tserver af730d19ec604080a589a5bd48fb7786 to finish bootstrapping
I20240502 01:57:58.536550 1050 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 47e0a5214cbc4a15b6b8aacd66ccb60c to finish bootstrapping
I20240502 01:57:58.541014 1050 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 778cc9c9c52648c28e0f04ccf879d34b to finish bootstrapping
I20240502 01:57:58.545610 1593 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.1.6.190:44199/ipki-ca-cert-der
http://127.1.6.190:44199/metrics
http://127.1.6.129:46371/threadz?group=all
http://127.1.6.131:40365/transactions?tablet_id=e54c9f86c1b84a6f8c0a773e029392d3
http://127.1.6.131:40365/threadz?group=all
http://127.1.6.130:45833/transactions?tablet_id=e54c9f86c1b84a6f8c0a773e029392d3
http://127.1.6.130:45833/threadz
http://127.1.6.190:44199/stacks
http://127.1.6.129:46371/maintenance-manager
http://127.1.6.190:44199/ipki-ca-cert
http://127.1.6.131:40365/threadz
http://127.1.6.190:44199/mem-trackers
http://127.1.6.129:46371/stacks
http://127.1.6.129:46371/metrics
http://127.1.6.190:44199/dump-entities
http://127.1.6.131:40365/stacks
http://127.1.6.131:40365/tablets
http://127.1.6.131:40365/mem-trackers
http://127.1.6.130:45833/startup
http://127.1.6.130:45833/scans
http://127.1.6.129:46371/scans
http://127.1.6.190:44199/masters
http://127.1.6.131:40365/scans
http://127.1.6.129:46371/tablets
http://127.1.6.190:44199/ipki-ca-cert-pem
http://127.1.6.130:45833/tablets
http://127.1.6.130:45833/metrics
http://127.1.6.130:45833/maintenance-manager
http://127.1.6.129:46371/startup
http://127.1.6.131:40365/startup
http://127.1.6.190:44199/tables
http://127.1.6.130:45833/threadz?group=all
http://127.1.6.129:46371/transactions?tablet_id=e54c9f86c1b84a6f8c0a773e029392d3
http://127.1.6.131:40365/maintenance-manager
http://127.1.6.129:46371/threadz
http://127.1.6.190:44199/tablet-servers
http://127.1.6.129:46371/mem-trackers
http://127.1.6.130:45833/mem-trackers
http://127.1.6.131:40365/metrics
http://127.1.6.190:44199/threadz
http://127.1.6.130:45833/stacks
http://127.1.6.190:44199/threadz?group=all
http://127.1.6.190:44199/startup
I20240502 01:57:58.548648 1050 linked_list-test-util.h:401] Running for: 30.000s
I20240502 01:57:58.548702 1050 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 01:57:58.548719 1050 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 01:57:58.566306 1369 raft_consensus.cc:1261] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Refusing update from remote peer 778cc9c9c52648c28e0f04ccf879d34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.566311 1237 raft_consensus.cc:1261] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Refusing update from remote peer 778cc9c9c52648c28e0f04ccf879d34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.566778 1368 raft_consensus.cc:1261] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Refusing update from remote peer 778cc9c9c52648c28e0f04ccf879d34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.566781 1236 raft_consensus.cc:1261] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Refusing update from remote peer 778cc9c9c52648c28e0f04ccf879d34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.566881 1503 raft_consensus.cc:1261] T e54c9f86c1b84a6f8c0a773e029392d3 P 778cc9c9c52648c28e0f04ccf879d34b [term 1 FOLLOWER]: Refusing update from remote peer 47e0a5214cbc4a15b6b8aacd66ccb60c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.567296 1603 consensus_queue.cc:1035] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.567418 1575 consensus_queue.cc:1035] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.567396 1583 consensus_queue.cc:1035] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.567687 1603 consensus_queue.cc:1035] T a293fda3c5f24da2a05338b3bbcebe70 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.567919 1603 consensus_queue.cc:1035] T 37eeb952913c42f7b0118e2c508bdc46 P 778cc9c9c52648c28e0f04ccf879d34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.571295 1235 raft_consensus.cc:1261] T e54c9f86c1b84a6f8c0a773e029392d3 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Refusing update from remote peer 47e0a5214cbc4a15b6b8aacd66ccb60c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.571689 1583 consensus_queue.cc:1035] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:58.574837 1599 mvcc.cc:205] Tried to move back new op lower bound from 7023063361799188480 to 7023063360999903232. Current Snapshot: MvccSnapshot[applied={T|T < 7023063361799188480}]
I20240502 01:57:58.575768 1604 mvcc.cc:205] Tried to move back new op lower bound from 7023063361802752000 to 7023063361194229760. Current Snapshot: MvccSnapshot[applied={T|T < 7023063361802752000}]
I20240502 01:57:58.578111 1610 mvcc.cc:205] Tried to move back new op lower bound from 7023063361802752000 to 7023063361194229760. Current Snapshot: MvccSnapshot[applied={T|T < 7023063361802752000}]
I20240502 01:58:00.593688 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:00.594739 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:00.598189 1620 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 01:58:01.357633 1620 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f3212e37dcd google::LogMessage::Fail() at ??:0
@ 0x7f3212e3bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3212e377cc google::LogMessage::Flush() at ??:0
@ 0x7f3212e38f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f32142195b1 kudu::operator-() at ??:0
@ 0x7f32162fcfd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f32162e605b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f32162e5f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f32162bc9bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f3216ee8861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f3216ee793a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7f3216ef0149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7f3211db2d09 std::function<>::operator()() at ??:0
@ 0x7f3211dad107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f3211daca38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f3211dac123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f3211dd4b89 handle_request at ??:0
@ 0x7f3211dd726b worker_thread at ??:0
@ 0x7f3214ab66db start_thread at ??:0
@ 0x7f3211fee71f clone at ??:0
W20240502 01:58:01.551591 1553 meta_cache.cc:302] tablet a293fda3c5f24da2a05338b3bbcebe70: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: recv got EOF from 127.1.6.131:34361 (error 108)
W20240502 01:58:01.561939 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.568917 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.580003 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.587524 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.605264 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.615756 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.643115 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.657817 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.680079 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.695129 1329 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.695341 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.699322 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:01.700138 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:01.703054 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.711661 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.720072 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.726382 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.731544 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.735524 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.747795 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.757128 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.766912 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.770864 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.781055 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.791553 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.796710 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.809857 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.828101 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.838335 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.843631 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.844590 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.864288 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.881736 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.902259 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.918499 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:01.943387 1196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.943387 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.946337 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:01.966151 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.001462 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.017748 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.022246 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.031179 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.043637 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.099411 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.099411 1196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.105470 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.126996 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.134115 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.135257 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.187243 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.203101 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.217737 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.233795 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.239138 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.282254 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.315757 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.318315 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.334702 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.357959 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.386376 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.422639 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:02.431164 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:02.432042 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:02.442263 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.485453 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.503304 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.542475 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.573127 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.575197 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.583853 1553 meta_cache.cc:302] tablet a293fda3c5f24da2a05338b3bbcebe70: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:02.618271 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.627724 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.657280 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.670445 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.714501 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:02.760416 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.761996 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.810201 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:02.853111 1665 raft_consensus.cc:482] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 778cc9c9c52648c28e0f04ccf879d34b)
I20240502 01:58:02.853250 1665 raft_consensus.cc:504] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:58:02.853838 1664 raft_consensus.cc:482] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 778cc9c9c52648c28e0f04ccf879d34b)
I20240502 01:58:02.853998 1664 raft_consensus.cc:504] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:58:02.854892 1665 leader_election.cc:288] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361), 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713)
W20240502 01:58:02.855250 1170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.855609 1664 leader_election.cc:288] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361), af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239)
I20240502 01:58:02.855705 1236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786" is_pre_election: true
I20240502 01:58:02.855813 1236 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 1.
I20240502 01:58:02.856122 1305 leader_election.cc:302] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters:
W20240502 01:58:02.856209 1302 leader_election.cc:334] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.856276 1664 raft_consensus.cc:2781] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:02.856320 1664 raft_consensus.cc:482] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 778cc9c9c52648c28e0f04ccf879d34b)
I20240502 01:58:02.856371 1664 raft_consensus.cc:3037] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:02.857540 1664 raft_consensus.cc:504] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:58:02.857904 1664 leader_election.cc:288] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 election: Requested vote from peers 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361), af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239)
W20240502 01:58:02.858561 1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58194: Illegal state: replica af730d19ec604080a589a5bd48fb7786 is not leader of this config: current role FOLLOWER
W20240502 01:58:02.859320 1302 leader_election.cc:334] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.859529 1236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "af730d19ec604080a589a5bd48fb7786"
I20240502 01:58:02.859639 1236 raft_consensus.cc:3037] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:02.860020 1368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a293fda3c5f24da2a05338b3bbcebe70" candidate_uuid: "af730d19ec604080a589a5bd48fb7786" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" is_pre_election: true
W20240502 01:58:02.860090 1170 leader_election.cc:334] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.860181 1368 raft_consensus.cc:2379] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af730d19ec604080a589a5bd48fb7786 in current term 2: Already voted for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in this term.
I20240502 01:58:02.860674 1171 leader_election.cc:302] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af730d19ec604080a589a5bd48fb7786; no voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, 778cc9c9c52648c28e0f04ccf879d34b
I20240502 01:58:02.860738 1236 raft_consensus.cc:2454] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e0a5214cbc4a15b6b8aacd66ccb60c in term 2.
I20240502 01:58:02.861107 1665 raft_consensus.cc:2726] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:02.861129 1305 leader_election.cc:302] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters: 778cc9c9c52648c28e0f04ccf879d34b
I20240502 01:58:02.861284 1665 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 778cc9c9c52648c28e0f04ccf879d34b)
I20240502 01:58:02.861359 1665 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:02.861452 1664 raft_consensus.cc:2781] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:02.861662 1665 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
I20240502 01:58:02.861657 1664 raft_consensus.cc:686] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 LEADER]: Becoming Leader. State: Replica: 47e0a5214cbc4a15b6b8aacd66ccb60c, State: Running, Role: LEADER
I20240502 01:58:02.861862 1664 consensus_queue.cc:237] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } }
I20240502 01:58:02.862071 1368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "af730d19ec604080a589a5bd48fb7786" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" is_pre_election: true
I20240502 01:58:02.862185 1368 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af730d19ec604080a589a5bd48fb7786 in term 1.
I20240502 01:58:02.862543 1171 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters:
I20240502 01:58:02.862699 1665 raft_consensus.cc:2781] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:02.862753 1665 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 778cc9c9c52648c28e0f04ccf879d34b)
I20240502 01:58:02.862802 1665 raft_consensus.cc:3037] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:02.863694 1170 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.863711 1665 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:02.864027 1665 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 election: Requested vote from peers 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130:34713), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
I20240502 01:58:02.864331 1368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eeb952913c42f7b0118e2c508bdc46" candidate_uuid: "af730d19ec604080a589a5bd48fb7786" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c"
I20240502 01:58:02.864430 1368 raft_consensus.cc:3037] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:02.864728 1093 catalog_manager.cc:5484] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c reported cstate change: term changed from 1 to 2, leader changed from 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131) to 47e0a5214cbc4a15b6b8aacd66ccb60c (127.1.6.130). New cstate: current_term: 2 leader_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:02.865136 1170 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:02.865396 1368 raft_consensus.cc:2454] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af730d19ec604080a589a5bd48fb7786 in term 2.
I20240502 01:58:02.865746 1171 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c, af730d19ec604080a589a5bd48fb7786; no voters: 778cc9c9c52648c28e0f04ccf879d34b
I20240502 01:58:02.866029 1665 raft_consensus.cc:2781] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:02.866253 1665 raft_consensus.cc:686] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [term 2 LEADER]: Becoming Leader. State: Replica: af730d19ec604080a589a5bd48fb7786, State: Running, Role: LEADER
I20240502 01:58:02.866539 1665 consensus_queue.cc:237] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 202, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:02.868744 1094 catalog_manager.cc:5484] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 reported cstate change: term changed from 1 to 2, leader changed from 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131) to af730d19ec604080a589a5bd48fb7786 (127.1.6.129). New cstate: current_term: 2 leader_uuid: "af730d19ec604080a589a5bd48fb7786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:02.914420 1236 raft_consensus.cc:1261] T a293fda3c5f24da2a05338b3bbcebe70 P af730d19ec604080a589a5bd48fb7786 [term 2 FOLLOWER]: Refusing update from remote peer 47e0a5214cbc4a15b6b8aacd66ccb60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20240502 01:58:02.914883 1670 consensus_queue.cc:1035] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
W20240502 01:58:02.916343 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:02.934420 1368 raft_consensus.cc:1261] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Refusing update from remote peer af730d19ec604080a589a5bd48fb7786: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
W20240502 01:58:02.935106 1170 consensus_peers.cc:485] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:02.935163 1665 consensus_queue.cc:1035] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240502 01:58:03.342574 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
I20240502 01:58:03.416193 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
I20240502 01:58:03.722904 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.380s user 0.267s sys 0.008s Metrics: {"bytes_written":147063,"cfile_init":1,"compiler_manager_pool.queue_time_us":309,"dirs.queue_time_us":13204,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":2064,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1455,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8032,"thread_start_us":264,"threads_started":2}
I20240502 01:58:03.724350 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 131638 bytes on disk
I20240502 01:58:03.730607 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 01:58:04.026316 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.610s user 0.280s sys 0.000s Metrics: {"bytes_written":148711,"cfile_init":1,"compiler_manager_pool.queue_time_us":24425,"dirs.queue_time_us":14006,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":993,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1641,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8131,"thread_start_us":187,"threads_started":2}
I20240502 01:58:04.027616 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 132492 bytes on disk
I20240502 01:58:04.028331 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 01:58:04.174722 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:04.428184 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:04.429353 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:05.275084 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:05.468467 1170 consensus_peers.cc:485] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:06.248565 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:06.249935 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:06.716912 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:58:06.717594 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:06.986485 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
I20240502 01:58:07.033783 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
I20240502 01:58:07.510215 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.476s user 0.264s sys 0.011s Metrics: {"bytes_written":11473,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":21474,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":953,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1807,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8218,"thread_start_us":71,"threads_started":1,"update_count":1771}
I20240502 01:58:07.511749 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 131706 bytes on disk
I20240502 01:58:07.512343 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 01:58:07.513197 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.014819
I20240502 01:58:07.545929 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.559s user 0.413s sys 0.000s Metrics: {"bytes_written":10664,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11346,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":4123,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1984,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8188,"thread_start_us":88,"threads_started":1,"update_count":1642}
I20240502 01:58:07.547417 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.014820
W20240502 01:58:07.711578 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:07.899216 1170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 01:58:07.907677 1170 consensus_peers.cc:485] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:07.988220 1595 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.000s sys 0.000s
I20240502 01:58:08.005962 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:08.006963 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:08.127030 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.613s user 0.438s sys 0.000s Metrics: {"bytes_written":297814,"cfile_cache_hit":13,"cfile_cache_hit_bytes":96224,"cfile_cache_miss":59,"cfile_cache_miss_bytes":970460,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":12036,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":5162,"drs_written":1,"lbm_read_time_us":3657,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":90,"lbm_write_time_us":2467,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":512495,"rows_written":16349,"thread_start_us":505,"threads_started":8}
I20240502 01:58:08.128090 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 300970 bytes on disk
I20240502 01:58:08.128490 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 01:58:08.211089 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.663s user 0.575s sys 0.003s Metrics: {"bytes_written":295789,"cfile_cache_hit":13,"cfile_cache_hit_bytes":94956,"cfile_cache_miss":59,"cfile_cache_miss_bytes":964660,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":4762,"dirs.run_cpu_time_us":546,"dirs.run_wall_time_us":8588,"drs_written":1,"lbm_read_time_us":2450,"lbm_reads_lt_1ms":95,"lbm_write_time_us":2943,"lbm_writes_lt_1ms":51,"mutex_wait_us":2026,"num_input_rowsets":2,"peak_mem_usage":509971,"rows_written":16220,"thread_start_us":779,"threads_started":7}
I20240502 01:58:08.211957 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 298480 bytes on disk
I20240502 01:58:08.212441 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 01:58:08.550009 1050 linked_list-test-util.h:417] Sample at HT timestamp: 7023063402681700352 Inserted count: 29150
W20240502 01:58:09.125219 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:09.947942 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:09.949065 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:10.217774 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:10.399497 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
I20240502 01:58:10.467181 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.000000
W20240502 01:58:10.702708 1170 consensus_peers.cc:485] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:10.708555 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.309s user 0.223s sys 0.000s Metrics: {"bytes_written":152861,"cfile_init":1,"dirs.queue_time_us":12027,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":5861,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1402,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8401,"thread_start_us":82,"threads_started":1}
I20240502 01:58:10.709246 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 143763 bytes on disk
I20240502 01:58:10.709674 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 01:58:10.710175 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.014699
I20240502 01:58:10.783953 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: FlushMRSOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.317s user 0.222s sys 0.008s Metrics: {"bytes_written":161436,"cfile_init":1,"dirs.queue_time_us":14403,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":2797,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1580,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":8694,"thread_start_us":75,"threads_started":1}
I20240502 01:58:10.784874 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 148218 bytes on disk
I20240502 01:58:10.785669 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":317,"lbm_reads_lt_1ms":4}
I20240502 01:58:10.786286 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=1.014696
I20240502 01:58:11.273506 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.563s user 0.540s sys 0.008s Metrics: {"bytes_written":450237,"cfile_cache_hit":10,"cfile_cache_hit_bytes":792,"cfile_cache_miss":78,"cfile_cache_miss_bytes":1664642,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":834,"drs_written":1,"lbm_read_time_us":5954,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":105,"lbm_write_time_us":3275,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":806339,"rows_written":24750,"thread_start_us":409,"threads_started":6}
I20240502 01:58:11.274349 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 459502 bytes on disk
I20240502 01:58:11.274833 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 01:58:11.656999 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: CompactRowSetsOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.871s user 0.667s sys 0.000s Metrics: {"bytes_written":5739,"cfile_cache_hit":11,"cfile_cache_hit_bytes":51809,"cfile_cache_miss":77,"cfile_cache_miss_bytes":1622423,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":3067,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":847,"drs_written":1,"lbm_read_time_us":3078,"lbm_reads_lt_1ms":105,"lbm_write_time_us":3122,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":808891,"reinsert_count":0,"rows_written":24914,"thread_start_us":577,"threads_started":7,"update_count":866}
I20240502 01:58:11.658213 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling MajorDeltaCompactionOp(a293fda3c5f24da2a05338b3bbcebe70): perf score=0.014666
I20240502 01:58:11.691310 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: MajorDeltaCompactionOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.033s user 0.032s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":8044,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":55,"lbm_read_time_us":270,"lbm_reads_lt_1ms":24,"lbm_write_time_us":243,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":866}
I20240502 01:58:11.691865 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70): 461175 bytes on disk
I20240502 01:58:11.692250 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(a293fda3c5f24da2a05338b3bbcebe70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
W20240502 01:58:11.705960 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:11.870138 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 01:58:12.061597 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:12.062323 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:12.882201 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:13.244720 1170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:58:13.245472 1170 consensus_peers.cc:485] T 37eeb952913c42f7b0118e2c508bdc46 P af730d19ec604080a589a5bd48fb7786 -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:13.381798 1789 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 38 ops behind the committed index
I20240502 01:58:13.448007 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling FlushMRSOp(37eeb952913c42f7b0118e2c508bdc46): perf score=1.000000
I20240502 01:58:13.529477 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling FlushMRSOp(37eeb952913c42f7b0118e2c508bdc46): perf score=1.000000
W20240502 01:58:13.884143 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:13.958308 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: FlushMRSOp(37eeb952913c42f7b0118e2c508bdc46) complete. Timing: real 0.429s user 0.225s sys 0.004s Metrics: {"bytes_written":141433,"cfile_init":1,"dirs.queue_time_us":13561,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":17536,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1513,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7691,"thread_start_us":71,"threads_started":1}
I20240502 01:58:13.958994 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling FlushMRSOp(e54c9f86c1b84a6f8c0a773e029392d3): perf score=1.000000
I20240502 01:58:13.959102 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: FlushMRSOp(37eeb952913c42f7b0118e2c508bdc46) complete. Timing: real 0.511s user 0.279s sys 0.008s Metrics: {"bytes_written":140601,"cfile_init":1,"dirs.queue_time_us":11984,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3541,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":7642,"thread_start_us":79,"threads_started":1}
I20240502 01:58:13.962442 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling FlushMRSOp(e54c9f86c1b84a6f8c0a773e029392d3): perf score=1.000000
I20240502 01:58:14.143864 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:14.147992 1598 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 01:58:14.237246 1598 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fc8c3f89dcd google::LogMessage::Fail() at ??:0
@ 0x7fc8c3f8db93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc8c3f897cc google::LogMessage::Flush() at ??:0
I20240502 01:58:14.270665 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: FlushMRSOp(e54c9f86c1b84a6f8c0a773e029392d3) complete. Timing: real 0.308s user 0.242s sys 0.012s Metrics: {"bytes_written":144495,"cfile_init":1,"dirs.queue_time_us":35,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1525,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7870}
I20240502 01:58:14.271351 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(e54c9f86c1b84a6f8c0a773e029392d3): 144716 bytes on disk
I20240502 01:58:14.271301 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: FlushMRSOp(e54c9f86c1b84a6f8c0a773e029392d3) complete. Timing: real 0.312s user 0.221s sys 0.000s Metrics: {"bytes_written":144379,"cfile_init":1,"dirs.queue_time_us":35,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1445,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7861}
I20240502 01:58:14.271800 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(e54c9f86c1b84a6f8c0a773e029392d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 01:58:14.271876 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(e54c9f86c1b84a6f8c0a773e029392d3): 144613 bytes on disk
I20240502 01:58:14.272256 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(e54c9f86c1b84a6f8c0a773e029392d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 01:58:14.272394 1418 maintenance_manager.cc:419] P 47e0a5214cbc4a15b6b8aacd66ccb60c: Scheduling UndoDeltaBlockGCOp(37eeb952913c42f7b0118e2c508bdc46): 142919 bytes on disk
I20240502 01:58:14.272768 1285 maintenance_manager.cc:419] P af730d19ec604080a589a5bd48fb7786: Scheduling UndoDeltaBlockGCOp(37eeb952913c42f7b0118e2c508bdc46): 143453 bytes on disk
I20240502 01:58:14.272890 1306 maintenance_manager.cc:643] P 47e0a5214cbc4a15b6b8aacd66ccb60c: UndoDeltaBlockGCOp(37eeb952913c42f7b0118e2c508bdc46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4}
I20240502 01:58:14.273293 1174 maintenance_manager.cc:643] P af730d19ec604080a589a5bd48fb7786: UndoDeltaBlockGCOp(37eeb952913c42f7b0118e2c508bdc46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
@ 0x7fc8c3f8af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc8c536b5b1 kudu::operator-() at ??:0
@ 0x7fc8c744efd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fc8c743805b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fc8c7437f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fc8c740e9bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fc8c803a861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fc8c803993a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fc8c8042149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fc8c2f04d09 std::function<>::operator()() at ??:0
@ 0x7fc8c2eff107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fc8c2efea38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fc8c2efe123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fc8c2f26b89 handle_request at ??:0
@ 0x7fc8c2f2926b worker_thread at ??:0
@ 0x7fc8c5c086db start_thread at ??:0
@ 0x7fc8c314071f clone at ??:0
W20240502 01:58:14.540145 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
W20240502 01:58:14.551352 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.570303 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.598577 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.629813 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:14.635839 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.683647 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.739198 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.749451 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:14.800904 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.877720 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:14.959780 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.056932 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.163681 1327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.231935 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.243275 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.265367 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.275585 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.294721 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.350497 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.351794 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:15.389885 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:15.406800 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.448472 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.461087 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:15.496004 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.548089 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:15.563731 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.572990 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.660183 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.688681 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:15.744405 1838 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:15.744602 1838 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:15.745255 1838 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:15.746444 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:15.746544 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:15.746665 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:15.746886 1838 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:15.750162 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.817597 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.885888 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.956588 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:15.997107 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.080998 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.164449 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.181751 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.301811 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.346583 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:16.420892 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:16.432606 1838 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:16.444504 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.535349 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:16.539812 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.548229 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:16.599004 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.710155 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.805971 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.875344 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:16.990682 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.003803 1305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 01:58:17.004510 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:17.078902 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.174607 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.299098 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.299098 1326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:17.385095 1838 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:17.385244 1838 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:17.385712 1838 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:17.386813 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:17.386942 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:17.386992 1305 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:17.387125 1838 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:17.398833 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:17.432529 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.478758 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:17.526234 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.567492 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:17.636337 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.760627 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.851033 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:17.863845 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:18.001206 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.074791 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.230808 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.346796 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:18.396719 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:18.469040 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.611685 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.630514 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:18.715204 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.889045 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:18.973800 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:19.006378 1838 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:19.006503 1838 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:19.006932 1838 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:19.008188 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:19.008437 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:19.008498 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:19.008639 1838 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:19.130888 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:19.146842 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:19.180141 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:19.271976 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:19.397987 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:19.419569 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:19.569587 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:19.672335 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:19.697041 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:19.783172 1838 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:58:19.876597 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:19.985183 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:19.994743 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:20.184870 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:20.303287 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:20.367251 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:20.391767 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:20.489238 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:20.632860 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:20.697707 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:20.801352 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:20.965703 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:21.121987 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:21.312705 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:21.394937 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:21.453855 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:21.556372 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:21.656734 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:21.665743 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:21.786773 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:21.792763 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:22.026150 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:22.031201 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:22.080097 1305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 01:58:22.279527 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:22.403726 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:22.462867 1838 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:22.463016 1838 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:22.463534 1838 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:22.464684 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:22.465063 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:22.465137 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:22.465322 1838 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:22.499454 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:22.527035 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:22.658421 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:22.708662 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:22.788403 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:22.864385 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:23.046695 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:23.165715 1838 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:58:23.243100 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:23.411371 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:23.443608 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:23.628229 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:23.848621 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:23.890805 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:23.892949 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:23.927783 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:24.023574 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:24.263803 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:24.401422 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:24.428879 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:24.691048 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:24.831128 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:24.840703 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:24.918967 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:24.979355 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:25.126736 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:25.260962 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:25.389304 1838 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:25.389432 1838 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:25.389844 1838 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:25.390868 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:25.391024 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:25.391127 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:25.391283 1838 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:25.400354 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:25.543984 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:25.713796 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:25.980304 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:26.015192 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:26.167122 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:26.250299 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:26.258023 1838 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
I20240502 01:58:26.408980 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:26.424937 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:26.500550 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:58:26.626304 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:26.879032 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:27.031837 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:27.097514 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:27.183387 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240502 01:58:27.184294 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:58:27.265772 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:27.341277 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:27.403784 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:27.579201 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:27.815297 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:28.065908 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:28.133412 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:28.294524 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:28.403347 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:28.567224 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:28.785985 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:28.840260 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:28.869107 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:58:29.072185 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:29.241163 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:29.284163 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:29.399950 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:29.587473 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:29.591682 1838 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:58:29.604132 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:58:29.622110 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:29.793180 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:30.113914 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:30.290370 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:30.308547 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:30.414702 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:30.650271 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:30.835916 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:31.180553 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:31.373240 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:31.376173 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:31.398828 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:31.411265 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:58:31.553727 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:31.745381 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:31.969894 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:58:32.100502 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:32.196740 1305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:58:32.197495 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:32.311280 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:32.404886 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:32.657660 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:32.691452 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:32.882467 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:32.939041 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
I20240502 01:58:33.195916 1848 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:33.196084 1848 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:33.196594 1848 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:33.197790 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:33.198201 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:33.198290 1305 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:33.198441 1848 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:33.224849 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:33.404866 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:33.463500 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:33.498294 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:33.799947 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:33.858131 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:33.994875 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:58:34.057220 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:34.385046 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:34.392267 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:34.436031 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:58:34.609767 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:34.657114 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:34.859054 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:34.977651 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:35.262070 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:35.392697 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:35.407510 1848 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:35.407646 1848 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:35.408097 1848 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:35.409132 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:35.409334 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:35.409505 1305 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:35.409706 1848 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:35.574658 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:35.872793 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:35.975159 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:35.979605 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:35.994866 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:36.184657 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:36.398698 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:36.501622 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:36.653337 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:58:36.800621 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:36.915928 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:58:37.009582 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:37.071394 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:37.138123 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:37.393608 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:37.426357 1305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240502 01:58:37.429056 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:37.782934 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:38.062440 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:38.215643 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:38.409935 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:58:38.412984 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:38.435248 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:38.710695 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:39.018676 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:58:39.079392 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:39.092394 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:39.316514 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:39.365161 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:39.400678 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:39.662559 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:58:39.761624 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:39.817229 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:58:40.031296 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:40.409355 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:40.440474 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:40.478366 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:40.702533 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:40.752539 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:58:41.127969 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:41.385213 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:41.403080 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:41.539572 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:58:41.592149 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:41.826287 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:42.075527 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:42.211463 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:58:42.232462 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:42.238477 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:58:42.407840 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:42.534927 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:42.586604 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:58:42.770982 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:42.773855 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:43.242911 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:43.246840 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:43.405999 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:43.730871 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:43.956984 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:44.175704 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:58:44.199286 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:44.302999 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:58:44.416887 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:44.458901 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:44.787169 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:58:44.928637 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:45.171610 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:45.171942 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:58:45.199297 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:45.410231 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
I20240502 01:58:45.415102 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:45.664856 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:45.943933 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:46.194463 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:46.408885 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:46.424530 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:46.552857 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:58:46.700681 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:46.923650 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:58:47.160202 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:47.209754 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:47.413064 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:47.463831 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:47.620878 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240502 01:58:47.621644 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:58:47.761480 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:58:47.927466 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:48.241183 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:48.417970 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:48.438659 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:48.524663 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:48.697046 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:49.019582 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:49.178829 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:58:49.333318 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:58:49.409404 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:49.477046 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:49.547772 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:49.810840 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:50.075860 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:58:50.269110 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:50.413081 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:50.476377 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:58:50.612140 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:50.799409 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:51.068701 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:51.404628 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:51.423112 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:51.598861 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:58:51.771178 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:58:51.875355 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:51.966579 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:52.068219 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:58:52.241983 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:52.407480 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:52.688541 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:52.691216 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 01:58:52.717553 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:58:52.944622 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:58:53.070299 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:53.239091 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:58:53.413003 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:53.515202 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:53.909075 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:54.348680 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:54.414564 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:54.477758 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:54.575124 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:58:54.579444 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:58:54.759891 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:54.840241 1848 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:58:54.840369 1848 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:58:54.840823 1848 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:58:54.842077 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:54.842335 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:58:54.842403 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:58:54.842530 1848 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:55.166114 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:58:55.171717 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:58:55.188562 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:55.277420 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:58:55.402158 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:55.616014 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:55.757994 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:56.042297 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:56.406236 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:56.479449 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:56.699252 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:58:56.770635 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:56.905683 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:57.223443 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:58:57.355242 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:57.402076 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:57.518262 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:58:57.775178 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:57.909715 1305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 01:58:57.910362 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:58:57.944734 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:58:58.238499 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:58.406741 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:58:58.648170 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:58:58.652808 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:58.946020 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:58:59.104979 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:58:59.132699 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:58:59.401573 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:58:59.536901 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:58:59.638156 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:58:59.855965 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:00.035285 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:00.131243 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:59:00.367609 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:59:00.406474 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:00.431169 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:00.946080 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:01.250454 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:01.334242 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:01.410313 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:01.591842 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:59:01.787832 1848 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:59:01.867337 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:02.245472 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:02.250587 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:02.352869 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:59:02.393538 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:02.485329 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:59:02.766719 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:02.797415 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:02.942400 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:59:03.171032 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:03.402926 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:03.732106 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:03.792183 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:59:03.908572 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:04.105638 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:04.398888 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:04.681052 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:04.755640 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:04.776881 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:05.000041 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:05.050525 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:05.164764 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:59:05.269593 1860 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
I20240502 01:59:05.405833 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:05.636207 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:06.003252 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:06.278999 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:59:06.417156 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:06.571384 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:06.603571 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:06.967764 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:07.205330 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:07.224781 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:07.288470 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:59:07.401376 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:07.574724 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:07.681327 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:07.936324 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:08.040107 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 01:59:08.410702 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:08.559787 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:08.586153 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
I20240502 01:59:08.625550 1860 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 6 similar messages]
I20240502 01:59:08.782090 1860 raft_consensus.cc:482] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af730d19ec604080a589a5bd48fb7786)
I20240502 01:59:08.782274 1860 raft_consensus.cc:504] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af730d19ec604080a589a5bd48fb7786" member_type: VOTER last_known_addr { host: "127.1.6.129" port: 39239 } } peers { permanent_uuid: "47e0a5214cbc4a15b6b8aacd66ccb60c" member_type: VOTER last_known_addr { host: "127.1.6.130" port: 34713 } } peers { permanent_uuid: "778cc9c9c52648c28e0f04ccf879d34b" member_type: VOTER last_known_addr { host: "127.1.6.131" port: 34361 } }
I20240502 01:59:08.782909 1860 leader_election.cc:288] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239), 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361)
W20240502 01:59:08.784104 1305 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:59:08.784204 1302 leader_election.cc:334] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
I20240502 01:59:08.784310 1302 leader_election.cc:302] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e0a5214cbc4a15b6b8aacd66ccb60c; no voters: 778cc9c9c52648c28e0f04ccf879d34b, af730d19ec604080a589a5bd48fb7786
I20240502 01:59:08.784492 1860 raft_consensus.cc:2726] T 37eeb952913c42f7b0118e2c508bdc46 P 47e0a5214cbc4a15b6b8aacd66ccb60c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:08.913169 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:09.235632 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:59:09.413632 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:09.526010 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:09.552913 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:09.670464 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:09.882498 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:09.900844 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:10.162261 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:59:10.409570 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:10.548354 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:10.882872 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:10.895284 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:11.406512 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:11.557857 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:11.640089 1860 consensus_queue.cc:786] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c [LEADER]: Peer 778cc9c9c52648c28e0f04ccf879d34b is lagging by at least 97 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:59:11.749627 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:11.870438 1302 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:11.897714 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:11.899847 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:12.250905 1302 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361): Couldn't send request to peer 778cc9c9c52648c28e0f04ccf879d34b. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:59:12.422986 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:12.575728 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:12.917730 1553 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica 778cc9c9c52648c28e0f04ccf879d34b (127.1.6.131:34361) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111)
W20240502 01:59:13.005159 1305 consensus_peers.cc:485] T e54c9f86c1b84a6f8c0a773e029392d3 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:13.045783 1302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.6.131:34361: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 01:59:13.257123 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
I20240502 01:59:13.414999 1626 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:59:13.605500 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:13.941076 1556 meta_cache.cc:302] tablet 37eeb952913c42f7b0118e2c508bdc46: replica af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239) has failed: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111)
W20240502 01:59:14.163681 1305 consensus_peers.cc:485] T a293fda3c5f24da2a05338b3bbcebe70 P 47e0a5214cbc4a15b6b8aacd66ccb60c -> Peer af730d19ec604080a589a5bd48fb7786 (127.1.6.129:39239): Couldn't send request to peer af730d19ec604080a589a5bd48fb7786. Status: Network error: Client connection negotiation failed: client connection to 127.1.6.129:39239: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:14.282495 1328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41034: Illegal state: replica 47e0a5214cbc4a15b6b8aacd66ccb60c is not leader of this config: current role FOLLOWER
W20240502 01:59:14.289963 1553 batcher.cc:441] Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290369 1050 client-test-util.cc:49] 8 failed ops. First 10 errors follow
I20240502 01:59:14.290458 1050 client-test-util.cc:55] Op INSERT int64 rand_key=4419211548827910150, int64 link_to=7612882695587954694, int64 insert_ts=1714615094229851 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290637 1050 client-test-util.cc:55] Op INSERT int64 rand_key=4634263679259836424, int64 link_to=2846584475987738632, int64 insert_ts=1714615094229925 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290678 1050 client-test-util.cc:55] Op INSERT int64 rand_key=5598641624002985999, int64 link_to=4049018202311950351, int64 insert_ts=1714615094230205 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290733 1050 client-test-util.cc:55] Op INSERT int64 rand_key=5813693754434912273, int64 link_to=-716857804823265263, int64 insert_ts=1714615094230279 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290783 1050 client-test-util.cc:55] Op INSERT int64 rand_key=4183305879332716577, int64 link_to=-6542195952547528671, int64 insert_ts=1714615094230800 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290828 1050 client-test-util.cc:55] Op INSERT int64 rand_key=4398358009764642851, int64 link_to=7138531376538517539, int64 insert_ts=1714615094230864 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290879 1050 client-test-util.cc:55] Op INSERT int64 rand_key=5362735954507792426, int64 link_to=8340965102862729258, int64 insert_ts=1714615094231088 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
I20240502 01:59:14.290920 1050 client-test-util.cc:55] Op INSERT int64 rand_key=5577788084939718700, int64 link_to=3575089095727513644, int64 insert_ts=1714615094231148 had status Timed out: Failed to write batch of 8 ops to tablet 37eeb952913c42f7b0118e2c508bdc46 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 37eeb952913c42f7b0118e2c508bdc46, num_ops: 8, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.1.6.129:39239 (error 108)
F20240502 01:59:14.290984 1050 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714615154 (unix time) try "date -d @1714615154" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000041a) received by PID 1050 (TID 0x7fb86c029280) from PID 1050; stack trace: ***
@ 0x7fb878340980 (unknown) at ??:0
@ 0x7fb875fc4fb7 gsignal at ??:0
@ 0x7fb875fc6921 abort at ??:0
@ 0x7fb876ed1dcd google::LogMessage::Fail() at ??:0
@ 0x7fb876ed5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb876ed17cc google::LogMessage::Flush() at ??:0
@ 0x7fb876ed2f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb8796037f2 kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x56500ac37236 kudu::client::FlushSessionOrDie() at ??:0
@ 0x56500ac3acaa kudu::LinkedListTester::LoadLinkedList() at ??:0
@ 0x56500ac304ca kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at ??:0
@ 0x7fb877dc80ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fb877dbcbdb testing::Test::Run() at ??:0
@ 0x7fb877dbcd9d testing::TestInfo::Run() at ??:0
@ 0x7fb877dbd377 testing::TestSuite::Run() at ??:0
@ 0x7fb877dbd77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fb877dc860d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fb877dbce63 testing::UnitTest::Run() at ??:0
@ 0x7fb8796151de RUN_ALL_TESTS() at ??:0
@ 0x7fb87961453d main at ??:0
@ 0x7fb875fa7bf7 __libc_start_main at ??:0
@ 0x56500ac2e34a _start at ??:0