[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 02:12:28.512112 817 test_util.cc:268] Using random seed: 78785347
I20240501 02:12:28.523715 817 linked_list-test.cc:100] Linked List Test Configuration:
I20240501 02:12:28.523804 817 linked_list-test.cc:101] --------------
I20240501 02:12:28.523846 817 linked_list-test.cc:102] 50 chains
I20240501 02:12:28.523912 817 linked_list-test.cc:103] 3 tablets
I20240501 02:12:28.523977 817 linked_list-test.cc:104] Mutations on
I20240501 02:12:28.524031 817 linked_list-test.cc:105] --------------
I20240501 02:12:28.524272 817 ts_itest-base.cc:112] Starting cluster with:
I20240501 02:12:28.524348 817 ts_itest-base.cc:113] --------------
I20240501 02:12:28.524408 817 ts_itest-base.cc:114] 3 tablet servers
I20240501 02:12:28.524463 817 ts_itest-base.cc:115] 3 replicas per TS
I20240501 02:12:28.524524 817 ts_itest-base.cc:116] --------------
2024-05-01T02:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:28Z Disabled control of system clock
I20240501 02:12:28.549283 817 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
/tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.126:36191
--webserver_interface=127.0.204.126
--webserver_port=0
--builtin_ntp_servers=127.0.204.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.126:36191
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240501 02:12:28.869779 829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:28.870101 829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:28.870256 829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:28.879196 829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:28.879318 829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:28.879369 829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:28.879454 829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:28.888350 829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.204.126:36191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.126:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.0.204.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 01 May 2024 01:43:23 UTC on efd1337a2aa3
build id 67
ASAN enabled
I20240501 02:12:28.889736 829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:28.891446 829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:28.902338 834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.903007 835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.903522 837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:28.904155 829 server_base.cc:1034] running on GCE node
I20240501 02:12:28.905010 829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:28.906491 829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:28.907723 829 hybrid_clock.cc:648] HybridClock initialized: now 1714529548907662 us; error 42 us; skew 500 ppm
I20240501 02:12:28.908067 829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 02:12:28.910431 829 webserver.cc:460] Webserver started at http://127.0.204.126:35571/ using document root <none> and password file <none>
I20240501 02:12:28.911168 829 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:28.911303 829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:28.911602 829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:28.914335 829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data/instance:
uuid: "dc1936c114004eda97ecfe413855b414"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9jfp"
I20240501 02:12:28.915102 829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal/instance:
uuid: "dc1936c114004eda97ecfe413855b414"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9jfp"
I20240501 02:12:28.920280 829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 02:12:28.923720 843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:28.925073 829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 02:12:28.925266 829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data,/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal
uuid: "dc1936c114004eda97ecfe413855b414"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9jfp"
I20240501 02:12:28.925493 829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:28.956117 829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:28.956933 829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:28.957197 829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:28.974855 829 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.126:36191
I20240501 02:12:28.974905 894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.126:36191 every 8 connection(s)
I20240501 02:12:28.976262 829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/data/info.pb
I20240501 02:12:28.979969 895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:28.986646 817 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu as pid 829
I20240501 02:12:28.986934 817 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/master-0/wal/instance
I20240501 02:12:28.990264 895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414: Bootstrap starting.
I20240501 02:12:28.994460 895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:28.996186 895 log.cc:824] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:29.004463 895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414: No bootstrap required, opened a new log
I20240501 02:12:29.010876 895 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } }
I20240501 02:12:29.011302 895 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:29.011427 895 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1936c114004eda97ecfe413855b414, State: Initialized, Role: FOLLOWER
I20240501 02:12:29.012089 895 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } }
I20240501 02:12:29.012355 895 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:29.012499 895 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:29.012691 895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:29.014956 895 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } }
I20240501 02:12:29.015555 895 leader_election.cc:302] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1936c114004eda97ecfe413855b414; no voters:
I20240501 02:12:29.016207 895 leader_election.cc:288] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:29.016355 900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:29.017560 900 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [term 1 LEADER]: Becoming Leader. State: Replica: dc1936c114004eda97ecfe413855b414, State: Running, Role: LEADER
I20240501 02:12:29.018308 895 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:29.018222 900 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } }
I20240501 02:12:29.022948 902 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1936c114004eda97ecfe413855b414. Latest consensus state: current_term: 1 leader_uuid: "dc1936c114004eda97ecfe413855b414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } } }
I20240501 02:12:29.023012 901 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1936c114004eda97ecfe413855b414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1936c114004eda97ecfe413855b414" member_type: VOTER last_known_addr { host: "127.0.204.126" port: 36191 } } }
I20240501 02:12:29.023296 902 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:29.023384 901 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:29.027416 907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:29.037154 907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:29.046188 907 catalog_manager.cc:1347] Generated new cluster ID: 5f305a4530544234b62266360c4c3ccc
I20240501 02:12:29.046315 907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:29.076493 907 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:29.077723 907 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:29.088353 907 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dc1936c114004eda97ecfe413855b414: Generated new TSK 0
I20240501 02:12:29.089179 907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:29.098268 817 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
/tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.65:0
--local_ip_for_outbound_sockets=127.0.204.65
--webserver_interface=127.0.204.65
--webserver_port=0
--tserver_master_addrs=127.0.204.126:36191
--builtin_ntp_servers=127.0.204.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240501 02:12:29.429718 919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:29.430022 919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:29.430183 919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:29.437559 919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:29.437734 919 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:12:29.437796 919 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:12:29.437860 919 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:12:29.437983 919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.65
I20240501 02:12:29.446913 919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.204.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.204.126:36191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.65
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 01 May 2024 01:43:23 UTC on efd1337a2aa3
build id 67
ASAN enabled
I20240501 02:12:29.448365 919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:29.450528 919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:29.463120 925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:29.463554 924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:29.464066 927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:29.464844 919 server_base.cc:1034] running on GCE node
I20240501 02:12:29.465502 919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:29.466478 919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:29.467708 919 hybrid_clock.cc:648] HybridClock initialized: now 1714529549467653 us; error 52 us; skew 500 ppm
I20240501 02:12:29.468008 919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 02:12:29.470432 919 webserver.cc:460] Webserver started at http://127.0.204.65:46505/ using document root <none> and password file <none>
I20240501 02:12:29.471163 919 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:29.471280 919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:29.471575 919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:29.474140 919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data/instance:
uuid: "30ff86a954ed44f79599a93fce287acc"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.474845 919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal/instance:
uuid: "30ff86a954ed44f79599a93fce287acc"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.480171 919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 02:12:29.483460 933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.484660 919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 02:12:29.484802 919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal
uuid: "30ff86a954ed44f79599a93fce287acc"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.484985 919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:29.515295 919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:29.516136 919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:29.516433 919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:29.517360 919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:29.518849 919 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:29.518941 919 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.519028 919 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:29.519081 919 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.549136 919 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.65:35425
I20240501 02:12:29.549208 1045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.65:35425 every 8 connection(s)
I20240501 02:12:29.550388 919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/data/info.pb
I20240501 02:12:29.557531 817 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu as pid 919
I20240501 02:12:29.557726 817 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-0/wal/instance
I20240501 02:12:29.561295 817 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
/tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.66:0
--local_ip_for_outbound_sockets=127.0.204.66
--webserver_interface=127.0.204.66
--webserver_port=0
--tserver_master_addrs=127.0.204.126:36191
--builtin_ntp_servers=127.0.204.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 02:12:29.562896 1046 heartbeater.cc:347] Connected to a master server at 127.0.204.126:36191
I20240501 02:12:29.563182 1046 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:29.563956 1046 heartbeater.cc:511] Master 127.0.204.126:36191 requested a full tablet report, sending...
I20240501 02:12:29.565997 860 ts_manager.cc:194] Registered new tserver with Master: 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425)
I20240501 02:12:29.567436 860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.65:57695
W20240501 02:12:29.883770 1050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:29.884065 1050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:29.884222 1050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:29.891333 1050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:29.891482 1050 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:12:29.891554 1050 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:12:29.891647 1050 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:12:29.891765 1050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.66
I20240501 02:12:29.900601 1050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.0.204.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.204.126:36191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.66
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 01 May 2024 01:43:23 UTC on efd1337a2aa3
build id 67
ASAN enabled
I20240501 02:12:29.901984 1050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:29.903714 1050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:29.915320 1055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:29.915833 1056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:29.916513 1058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:29.917145 1050 server_base.cc:1034] running on GCE node
I20240501 02:12:29.917773 1050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:29.918778 1050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:29.920028 1050 hybrid_clock.cc:648] HybridClock initialized: now 1714529549919984 us; error 38 us; skew 500 ppm
I20240501 02:12:29.920343 1050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 02:12:29.922570 1050 webserver.cc:460] Webserver started at http://127.0.204.66:42239/ using document root <none> and password file <none>
I20240501 02:12:29.923269 1050 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:29.923375 1050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:29.923653 1050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:29.926229 1050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data/instance:
uuid: "610de482e0714825868dd5b03c7e1f51"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.926975 1050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal/instance:
uuid: "610de482e0714825868dd5b03c7e1f51"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.932361 1050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 02:12:29.935734 1064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.937095 1050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 02:12:29.937288 1050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal
uuid: "610de482e0714825868dd5b03c7e1f51"
format_stamp: "Formatted at 2024-05-01 02:12:29 on dist-test-slave-9jfp"
I20240501 02:12:29.937556 1050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:29.956687 1050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:29.957544 1050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:29.957830 1050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:29.958750 1050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:29.960541 1050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:29.960630 1050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.960721 1050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:29.960789 1050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:29.994405 1050 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.66:42855
I20240501 02:12:29.994437 1176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.66:42855 every 8 connection(s)
I20240501 02:12:29.995704 1050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/data/info.pb
I20240501 02:12:29.998931 817 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu as pid 1050
I20240501 02:12:29.999159 817 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-1/wal/instance
I20240501 02:12:30.002629 817 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
/tmp/dist-test-task4Eqqgk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.67:0
--local_ip_for_outbound_sockets=127.0.204.67
--webserver_interface=127.0.204.67
--webserver_port=0
--tserver_master_addrs=127.0.204.126:36191
--builtin_ntp_servers=127.0.204.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 02:12:30.009254 1177 heartbeater.cc:347] Connected to a master server at 127.0.204.126:36191
I20240501 02:12:30.009562 1177 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:30.010344 1177 heartbeater.cc:511] Master 127.0.204.126:36191 requested a full tablet report, sending...
I20240501 02:12:30.011729 860 ts_manager.cc:194] Registered new tserver with Master: 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855)
I20240501 02:12:30.012485 860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.66:60015
W20240501 02:12:30.322103 1180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:30.322420 1180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:30.322602 1180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:30.329994 1180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:30.330166 1180 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:12:30.330227 1180 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:12:30.330325 1180 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:12:30.330442 1180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.67
I20240501 02:12:30.340116 1180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.0.204.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.204.126:36191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.67
--log_dir=/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 01 May 2024 01:43:23 UTC on efd1337a2aa3
build id 67
ASAN enabled
I20240501 02:12:30.341598 1180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:30.343437 1180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:30.355764 1186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:30.356266 1185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:30.356884 1188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:30.357615 1180 server_base.cc:1034] running on GCE node
I20240501 02:12:30.358242 1180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:30.359217 1180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:30.360452 1180 hybrid_clock.cc:648] HybridClock initialized: now 1714529550360398 us; error 50 us; skew 500 ppm
I20240501 02:12:30.360780 1180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 02:12:30.363121 1180 webserver.cc:460] Webserver started at http://127.0.204.67:39059/ using document root <none> and password file <none>
I20240501 02:12:30.363835 1180 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:30.363970 1180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:30.364271 1180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:30.366942 1180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data/instance:
uuid: "571678236b844d108b1227119d8e90d1"
format_stamp: "Formatted at 2024-05-01 02:12:30 on dist-test-slave-9jfp"
I20240501 02:12:30.367650 1180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal/instance:
uuid: "571678236b844d108b1227119d8e90d1"
format_stamp: "Formatted at 2024-05-01 02:12:30 on dist-test-slave-9jfp"
I20240501 02:12:30.372501 1180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240501 02:12:30.375780 1194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:30.377135 1180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 02:12:30.377347 1180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal
uuid: "571678236b844d108b1227119d8e90d1"
format_stamp: "Formatted at 2024-05-01 02:12:30 on dist-test-slave-9jfp"
I20240501 02:12:30.377553 1180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:30.407178 1180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:30.408000 1180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:30.408270 1180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:30.409158 1180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:30.410555 1180 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:30.410646 1180 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:30.410758 1180 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:30.410823 1180 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:30.444778 1180 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.67:43585
I20240501 02:12:30.444833 1306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.67:43585 every 8 connection(s)
I20240501 02:12:30.446208 1180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/data/info.pb
I20240501 02:12:30.449942 817 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4Eqqgk/build/asan/bin/kudu as pid 1180
I20240501 02:12:30.450160 817 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4Eqqgk/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714529548438045-817-0/linked-list-cluster/ts-2/wal/instance
I20240501 02:12:30.459497 1307 heartbeater.cc:347] Connected to a master server at 127.0.204.126:36191
I20240501 02:12:30.459780 1307 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:30.460536 1307 heartbeater.cc:511] Master 127.0.204.126:36191 requested a full tablet report, sending...
I20240501 02:12:30.461824 860 ts_manager.cc:194] Registered new tserver with Master: 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:30.462620 860 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.67:41629
I20240501 02:12:30.467190 817 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:30.490010 860 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39542:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240501 02:12:30.495896 860 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:12:30.537101 981 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:12:30.537278 1241 tablet_service.cc:1467] Processing CreateTablet for tablet 2340ca4359bc4d6c9f8630426f391f70 (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:12:30.538479 980 tablet_service.cc:1467] Processing CreateTablet for tablet 2340ca4359bc4d6c9f8630426f391f70 (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:12:30.537115 1242 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:12:30.539250 1241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340ca4359bc4d6c9f8630426f391f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.539701 980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340ca4359bc4d6c9f8630426f391f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.539705 1242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.537138 1240 tablet_service.cc:1467] Processing CreateTablet for tablet d3c31b5c984043f38cd869f5b21c067e (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:12:30.538717 979 tablet_service.cc:1467] Processing CreateTablet for tablet d3c31b5c984043f38cd869f5b21c067e (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:12:30.540608 1112 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:12:30.541810 979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c31b5c984043f38cd869f5b21c067e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.542377 1112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.542747 981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.544023 1240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c31b5c984043f38cd869f5b21c067e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.544775 1110 tablet_service.cc:1467] Processing CreateTablet for tablet d3c31b5c984043f38cd869f5b21c067e (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:12:30.544979 1111 tablet_service.cc:1467] Processing CreateTablet for tablet 2340ca4359bc4d6c9f8630426f391f70 (DEFAULT_TABLE table=TestTable [id=641382915785416f98355ba9fb028c37]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:12:30.546303 1110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c31b5c984043f38cd869f5b21c067e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.548991 1111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340ca4359bc4d6c9f8630426f391f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:30.565322 1326 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: Bootstrap starting.
I20240501 02:12:30.569110 1326 tablet_bootstrap.cc:654] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.570750 1046 heartbeater.cc:503] Master 127.0.204.126:36191 was elected leader, sending a full tablet report...
I20240501 02:12:30.572116 1325 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: Bootstrap starting.
I20240501 02:12:30.572326 1326 log.cc:824] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:30.575589 1326 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: No bootstrap required, opened a new log
I20240501 02:12:30.575635 1325 tablet_bootstrap.cc:654] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.576002 1326 ts_tablet_manager.cc:1367] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240501 02:12:30.576886 1325 log.cc:824] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:30.581568 1329 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: Bootstrap starting.
I20240501 02:12:30.585168 1329 tablet_bootstrap.cc:654] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.585973 1325 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: No bootstrap required, opened a new log
I20240501 02:12:30.586160 1329 log.cc:824] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:30.586266 1325 ts_tablet_manager.cc:1367] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20240501 02:12:30.586870 1326 raft_consensus.cc:348] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.587323 1326 raft_consensus.cc:374] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.587424 1326 raft_consensus.cc:729] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610de482e0714825868dd5b03c7e1f51, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.588053 1326 consensus_queue.cc:260] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.590924 1326 ts_tablet_manager.cc:1398] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20240501 02:12:30.591547 1329 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: No bootstrap required, opened a new log
I20240501 02:12:30.591794 1329 ts_tablet_manager.cc:1367] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 02:12:30.593684 1177 heartbeater.cc:503] Master 127.0.204.126:36191 was elected leader, sending a full tablet report...
I20240501 02:12:30.594537 1326 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: Bootstrap starting.
I20240501 02:12:30.594435 1325 raft_consensus.cc:348] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.594843 1325 raft_consensus.cc:374] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.594949 1325 raft_consensus.cc:729] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571678236b844d108b1227119d8e90d1, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.595643 1325 consensus_queue.cc:260] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.596324 1326 tablet_bootstrap.cc:654] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.599629 1329 raft_consensus.cc:348] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.600101 1329 raft_consensus.cc:374] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.600206 1329 raft_consensus.cc:729] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ff86a954ed44f79599a93fce287acc, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.600982 1329 consensus_queue.cc:260] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.602077 1326 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: No bootstrap required, opened a new log
I20240501 02:12:30.602226 1326 ts_tablet_manager.cc:1367] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 02:12:30.602313 1307 heartbeater.cc:503] Master 127.0.204.126:36191 was elected leader, sending a full tablet report...
I20240501 02:12:30.602854 1325 ts_tablet_manager.cc:1398] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1: Time spent starting tablet: real 0.016s user 0.003s sys 0.013s
I20240501 02:12:30.603155 1326 raft_consensus.cc:348] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.603309 1325 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1: Bootstrap starting.
I20240501 02:12:30.603356 1326 raft_consensus.cc:374] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.603432 1326 raft_consensus.cc:729] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610de482e0714825868dd5b03c7e1f51, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.603680 1326 consensus_queue.cc:260] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.604188 1326 ts_tablet_manager.cc:1398] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 02:12:30.604970 1325 tablet_bootstrap.cc:654] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.605918 1326 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51: Bootstrap starting.
I20240501 02:12:30.607937 1326 tablet_bootstrap.cc:654] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.613386 1325 tablet_bootstrap.cc:492] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1: No bootstrap required, opened a new log
I20240501 02:12:30.613579 1325 ts_tablet_manager.cc:1367] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 02:12:30.614486 1325 raft_consensus.cc:348] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.614702 1325 raft_consensus.cc:374] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.614799 1325 raft_consensus.cc:729] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571678236b844d108b1227119d8e90d1, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.615021 1326 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51: No bootstrap required, opened a new log
I20240501 02:12:30.615190 1326 ts_tablet_manager.cc:1367] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 02:12:30.615100 1325 consensus_queue.cc:260] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.615581 1329 ts_tablet_manager.cc:1398] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc: Time spent starting tablet: real 0.024s user 0.015s sys 0.008s
I20240501 02:12:30.615903 1326 raft_consensus.cc:348] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.616041 1329 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: Bootstrap starting.
I20240501 02:12:30.616142 1326 raft_consensus.cc:374] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.616227 1326 raft_consensus.cc:729] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610de482e0714825868dd5b03c7e1f51, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.616467 1325 ts_tablet_manager.cc:1398] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 02:12:30.616499 1326 consensus_queue.cc:260] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.616773 1325 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: Bootstrap starting.
I20240501 02:12:30.618183 1329 tablet_bootstrap.cc:654] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.618399 1325 tablet_bootstrap.cc:654] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.618978 1326 ts_tablet_manager.cc:1398] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 02:12:30.621840 1329 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: No bootstrap required, opened a new log
I20240501 02:12:30.622016 1329 ts_tablet_manager.cc:1367] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 02:12:30.622375 1325 tablet_bootstrap.cc:492] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: No bootstrap required, opened a new log
I20240501 02:12:30.622539 1325 ts_tablet_manager.cc:1367] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 02:12:30.622990 1329 raft_consensus.cc:348] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.623148 1329 raft_consensus.cc:374] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.623217 1329 raft_consensus.cc:729] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ff86a954ed44f79599a93fce287acc, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.623109 1325 raft_consensus.cc:348] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.623322 1325 raft_consensus.cc:374] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.623410 1325 raft_consensus.cc:729] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571678236b844d108b1227119d8e90d1, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.623445 1329 consensus_queue.cc:260] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.623616 1325 consensus_queue.cc:260] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.623973 1329 ts_tablet_manager.cc:1398] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 02:12:30.624140 1325 ts_tablet_manager.cc:1398] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 02:12:30.624209 1329 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc: Bootstrap starting.
I20240501 02:12:30.625502 1329 tablet_bootstrap.cc:654] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:30.627629 1329 tablet_bootstrap.cc:492] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc: No bootstrap required, opened a new log
I20240501 02:12:30.627781 1329 ts_tablet_manager.cc:1367] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 02:12:30.628302 1329 raft_consensus.cc:348] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.628479 1329 raft_consensus.cc:374] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:30.628552 1329 raft_consensus.cc:729] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ff86a954ed44f79599a93fce287acc, State: Initialized, Role: FOLLOWER
I20240501 02:12:30.628772 1329 consensus_queue.cc:260] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.629173 1329 ts_tablet_manager.cc:1398] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 02:12:30.638887 1332 raft_consensus.cc:482] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:30.639149 1332 raft_consensus.cc:504] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.640877 1332 leader_election.cc:288] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425)
I20240501 02:12:30.647998 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51" is_pre_election: true
I20240501 02:12:30.648550 1132 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 0.
I20240501 02:12:30.648634 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc" is_pre_election: true
I20240501 02:12:30.649116 1001 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 0.
I20240501 02:12:30.649346 1198 leader_election.cc:302] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571678236b844d108b1227119d8e90d1, 610de482e0714825868dd5b03c7e1f51; no voters:
I20240501 02:12:30.649758 1332 raft_consensus.cc:2781] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:30.649896 1332 raft_consensus.cc:482] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:30.649989 1332 raft_consensus.cc:3037] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.651939 1332 raft_consensus.cc:504] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.652508 1332 leader_election.cc:288] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 election: Requested vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425)
I20240501 02:12:30.652966 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51"
I20240501 02:12:30.653040 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc"
I20240501 02:12:30.653183 1132 raft_consensus.cc:3037] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.653210 1001 raft_consensus.cc:3037] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.656265 1001 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 1.
I20240501 02:12:30.656265 1132 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 1.
I20240501 02:12:30.656729 1198 leader_election.cc:302] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571678236b844d108b1227119d8e90d1, 610de482e0714825868dd5b03c7e1f51; no voters:
I20240501 02:12:30.657060 1332 raft_consensus.cc:2781] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:30.657562 1332 raft_consensus.cc:686] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [term 1 LEADER]: Becoming Leader. State: Replica: 571678236b844d108b1227119d8e90d1, State: Running, Role: LEADER
I20240501 02:12:30.658068 1332 consensus_queue.cc:237] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.662794 860 catalog_manager.cc:5484] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 571678236b844d108b1227119d8e90d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 571678236b844d108b1227119d8e90d1 (127.0.204.67). New cstate: current_term: 1 leader_uuid: "571678236b844d108b1227119d8e90d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } health_report { overall_health: HEALTHY } } }
W20240501 02:12:30.697960 1308 tablet.cc:2369] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:12:30.747718 1178 tablet.cc:2369] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:12:30.802793 1047 tablet.cc:2369] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:12:30.837643 1334 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:30.837831 1334 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.839501 1334 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:30.846213 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c31b5c984043f38cd869f5b21c067e" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51" is_pre_election: true
I20240501 02:12:30.846514 1132 raft_consensus.cc:2454] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 0.
I20240501 02:12:30.846540 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c31b5c984043f38cd869f5b21c067e" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571678236b844d108b1227119d8e90d1" is_pre_election: true
I20240501 02:12:30.846951 1262 raft_consensus.cc:2454] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 0.
I20240501 02:12:30.847003 937 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters:
I20240501 02:12:30.847419 1334 raft_consensus.cc:2781] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:30.847537 1334 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:30.847640 1334 raft_consensus.cc:3037] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.849722 1334 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.850368 1334 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 1 election: Requested vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:30.850697 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c31b5c984043f38cd869f5b21c067e" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51"
I20240501 02:12:30.850931 1132 raft_consensus.cc:3037] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.850870 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c31b5c984043f38cd869f5b21c067e" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571678236b844d108b1227119d8e90d1"
I20240501 02:12:30.851063 1262 raft_consensus.cc:3037] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.853852 1132 raft_consensus.cc:2454] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 1.
I20240501 02:12:30.853957 1262 raft_consensus.cc:2454] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 1.
I20240501 02:12:30.854300 937 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters:
I20240501 02:12:30.854636 1334 raft_consensus.cc:2781] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:30.855077 1334 raft_consensus.cc:686] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [term 1 LEADER]: Becoming Leader. State: Replica: 30ff86a954ed44f79599a93fce287acc, State: Running, Role: LEADER
I20240501 02:12:30.855573 1334 consensus_queue.cc:237] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.859501 860 catalog_manager.cc:5484] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ff86a954ed44f79599a93fce287acc (127.0.204.65). New cstate: current_term: 1 leader_uuid: "30ff86a954ed44f79599a93fce287acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } health_report { overall_health: UNKNOWN } } }
I20240501 02:12:30.895246 1332 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:30.895514 1332 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.896135 1332 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855)
I20240501 02:12:30.896713 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc" is_pre_election: true
I20240501 02:12:30.896728 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51" is_pre_election: true
I20240501 02:12:30.896947 1132 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 0.
I20240501 02:12:30.896965 1001 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 0.
I20240501 02:12:30.897409 1198 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571678236b844d108b1227119d8e90d1, 610de482e0714825868dd5b03c7e1f51; no voters:
I20240501 02:12:30.897672 1332 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:30.897763 1332 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:30.897842 1332 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.899610 1332 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.900156 1332 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 election: Requested vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855)
I20240501 02:12:30.900519 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc"
I20240501 02:12:30.900725 1001 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.900648 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "571678236b844d108b1227119d8e90d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51"
I20240501 02:12:30.900833 1132 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:30.902690 1001 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 1.
I20240501 02:12:30.902750 1132 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571678236b844d108b1227119d8e90d1 in term 1.
I20240501 02:12:30.903085 1197 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1; no voters:
I20240501 02:12:30.903344 1332 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:30.903499 1332 raft_consensus.cc:686] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [term 1 LEADER]: Becoming Leader. State: Replica: 571678236b844d108b1227119d8e90d1, State: Running, Role: LEADER
I20240501 02:12:30.903743 1332 consensus_queue.cc:237] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:30.905983 860 catalog_manager.cc:5484] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 571678236b844d108b1227119d8e90d1 (127.0.204.67). New cstate: current_term: 1 leader_uuid: "571678236b844d108b1227119d8e90d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } health_report { overall_health: HEALTHY } } }
I20240501 02:12:30.964162 817 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:30.966841 817 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 30ff86a954ed44f79599a93fce287acc to finish bootstrapping
I20240501 02:12:30.976640 817 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 610de482e0714825868dd5b03c7e1f51 to finish bootstrapping
I20240501 02:12:30.986786 817 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 571678236b844d108b1227119d8e90d1 to finish bootstrapping
I20240501 02:12:30.995281 1349 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.0.204.67:39059/mem-trackers
http://127.0.204.65:46505/tablets
http://127.0.204.126:35571/tables
http://127.0.204.67:39059/threadz?group=all
http://127.0.204.67:39059/transactions?tablet_id=6e9fb5b0cc2a40b58c067ccbc8b28b8f
http://127.0.204.65:46505/threadz?group=all
http://127.0.204.66:42239/scans
http://127.0.204.66:42239/threadz?group=all
http://127.0.204.67:39059/scans
http://127.0.204.67:39059/maintenance-manager
http://127.0.204.126:35571/dump-entities
http://127.0.204.65:46505/transactions?tablet_id=6e9fb5b0cc2a40b58c067ccbc8b28b8f
http://127.0.204.67:39059/threadz
http://127.0.204.66:42239/tablets
http://127.0.204.126:35571/ipki-ca-cert-pem
http://127.0.204.65:46505/startup
http://127.0.204.126:35571/stacks
http://127.0.204.67:39059/stacks
http://127.0.204.67:39059/tablets
http://127.0.204.65:46505/scans
http://127.0.204.126:35571/ipki-ca-cert-der
http://127.0.204.126:35571/threadz?group=all
http://127.0.204.126:35571/metrics
http://127.0.204.126:35571/startup
http://127.0.204.126:35571/masters
http://127.0.204.126:35571/threadz
http://127.0.204.65:46505/mem-trackers
http://127.0.204.66:42239/mem-trackers
http://127.0.204.66:42239/threadz
http://127.0.204.66:42239/transactions?tablet_id=6e9fb5b0cc2a40b58c067ccbc8b28b8f
http://127.0.204.67:39059/startup
http://127.0.204.126:35571/mem-trackers
http://127.0.204.66:42239/maintenance-manager
http://127.0.204.65:46505/maintenance-manager
http://127.0.204.66:42239/stacks
http://127.0.204.65:46505/threadz
http://127.0.204.126:35571/ipki-ca-cert
http://127.0.204.66:42239/startup
http://127.0.204.65:46505/metrics
http://127.0.204.66:42239/metrics
http://127.0.204.126:35571/tablet-servers
http://127.0.204.65:46505/stacks
http://127.0.204.67:39059/metrics
I20240501 02:12:30.999629 817 linked_list-test-util.h:401] Running for: 30.000s
I20240501 02:12:30.999847 817 linked_list-test-util.h:402] Sampling every 10000000 us
I20240501 02:12:30.999935 817 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240501 02:12:31.043148 1132 raft_consensus.cc:1261] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Refusing update from remote peer 30ff86a954ed44f79599a93fce287acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:31.043160 1262 raft_consensus.cc:1261] T d3c31b5c984043f38cd869f5b21c067e P 571678236b844d108b1227119d8e90d1 [term 1 FOLLOWER]: Refusing update from remote peer 30ff86a954ed44f79599a93fce287acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:31.044231 1334 consensus_queue.cc:1035] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:31.044899 1345 consensus_queue.cc:1035] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:31.049343 1131 raft_consensus.cc:1261] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Refusing update from remote peer 571678236b844d108b1227119d8e90d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
F20240501 02:12:31.049909 1350 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
I20240501 02:12:31.051378 1001 raft_consensus.cc:1261] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Refusing update from remote peer 571678236b844d108b1227119d8e90d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:31.050192 1332 consensus_queue.cc:1035] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:31.052997 1341 consensus_queue.cc:1035] T 2340ca4359bc4d6c9f8630426f391f70 P 571678236b844d108b1227119d8e90d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f4a3205adcd google::LogMessage::Fail() at ??:0
@ 0x7f4a3205eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4a3205a7cc google::LogMessage::Flush() at ??:0
@ 0x7f4a3205bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a34c83c80 kudu::operator-() at ??:0
@ 0x7f4a4190414a kudu::tablet::WriteOp::ToString() at ??:0
I20240501 02:12:31.071120 1359 mvcc.cc:205] Tried to move back new op lower bound from 7022713041061699584 to 7022713040307961856. Current Snapshot: MvccSnapshot[applied={T|T < 7022713041061699584}]
@ 0x7f4a418ce5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f4a418ce336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f4a4187b2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f4a44348d87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f4a4435793c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
I20240501 02:12:31.099076 1361 mvcc.cc:205] Tried to move back new op lower bound from 7022713041061699584 to 7022713040307961856. Current Snapshot: MvccSnapshot[applied={T|T < 7022713041061699584}]
@ 0x7f4a443577a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f4a3dd8d02a std::function<>::operator()() at ??:0
@ 0x7f4a3dd857e6 kudu::Webserver::RunPathHandler() at ??:0
I20240501 02:12:31.115124 1364 mvcc.cc:205] Tried to move back new op lower bound from 7022713041081430016 to 7022713040503746560. Current Snapshot: MvccSnapshot[applied={T|T < 7022713041081430016}]
@ 0x7f4a3dd807f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f4a3dd7e927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f4a3ddbc089 handle_request at ??:0
@ 0x7f4a3ddbe76b worker_thread at ??:0
@ 0x7f4a380cf6db start_thread at ??:0
@ 0x7f4a30f0d71f clone at ??:0
W20240501 02:12:31.295975 1311 meta_cache.cc:302] tablet 6e9fb5b0cc2a40b58c067ccbc8b28b8f: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: recv got EOF from 127.0.204.67:43585 (error 108)
W20240501 02:12:31.310034 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.315920 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:31.317845 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:31.328275 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.335186 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.358724 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.372179 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.399569 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.415954 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.451112 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.467329 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.511835 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.531014 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.590322 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.616147 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.639211 935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:31.640516 935 consensus_peers.cc:485] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:12:31.667622 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.697510 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.757090 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.785993 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.852663 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:31.854243 1352 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:31.886046 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:31.960155 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:31.998741 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.076376 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.114781 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.198391 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:32.232565 1399 raft_consensus.cc:482] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:32.232760 1399 raft_consensus.cc:504] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.234372 1399 leader_election.cc:288] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:32.235810 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:32.237504 1066 leader_election.cc:334] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:32.240824 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:32.241153 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "610de482e0714825868dd5b03c7e1f51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc" is_pre_election: true
I20240501 02:12:32.241441 1001 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 610de482e0714825868dd5b03c7e1f51 in term 1.
I20240501 02:12:32.242033 1067 leader_election.cc:302] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters: 571678236b844d108b1227119d8e90d1
I20240501 02:12:32.242841 1399 raft_consensus.cc:2781] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 02:12:32.242974 1399 raft_consensus.cc:482] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:32.243096 1399 raft_consensus.cc:3037] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:12:32.246114 1399 raft_consensus.cc:504] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.246944 1399 leader_election.cc:288] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: Requested vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:32.247685 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9fb5b0cc2a40b58c067ccbc8b28b8f" candidate_uuid: "610de482e0714825868dd5b03c7e1f51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc"
I20240501 02:12:32.247915 1001 raft_consensus.cc:3037] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Advancing to term 2
W20240501 02:12:32.248921 1066 leader_election.cc:334] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:32.250785 1001 raft_consensus.cc:2454] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 610de482e0714825868dd5b03c7e1f51 in term 2.
I20240501 02:12:32.251287 1067 leader_election.cc:302] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters: 571678236b844d108b1227119d8e90d1
I20240501 02:12:32.251637 1399 raft_consensus.cc:2781] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:12:32.252084 1399 raft_consensus.cc:686] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [term 2 LEADER]: Becoming Leader. State: Replica: 610de482e0714825868dd5b03c7e1f51, State: Running, Role: LEADER
I20240501 02:12:32.252580 1399 consensus_queue.cc:237] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.256474 860 catalog_manager.cc:5484] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 reported cstate change: term changed from 1 to 2, leader changed from 571678236b844d108b1227119d8e90d1 (127.0.204.67) to 610de482e0714825868dd5b03c7e1f51 (127.0.204.66). New cstate: current_term: 2 leader_uuid: "610de482e0714825868dd5b03c7e1f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } health_report { overall_health: UNKNOWN } } }
I20240501 02:12:32.289345 1001 raft_consensus.cc:1261] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Refusing update from remote peer 610de482e0714825868dd5b03c7e1f51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 02:12:32.290098 1404 consensus_queue.cc:1035] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:12:32.290410 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:12:32.303997 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:32.309985 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.315289 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.332547 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.341811 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.359907 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.372231 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.402812 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.417979 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.454056 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.470186 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.511512 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.531630 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.578264 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:32.580567 1345 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 571678236b844d108b1227119d8e90d1)
I20240501 02:12:32.580677 1418 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 571678236b844d108b1227119d8e90d1)
I20240501 02:12:32.580797 1345 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.580842 1418 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.581688 1345 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:32.581840 1418 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:32.584517 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:32.584529 935 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:32.584880 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51" is_pre_election: true
I20240501 02:12:32.584961 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "610de482e0714825868dd5b03c7e1f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc" is_pre_election: true
I20240501 02:12:32.585093 1132 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 1.
I20240501 02:12:32.585202 1001 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 610de482e0714825868dd5b03c7e1f51 in term 1.
I20240501 02:12:32.585537 937 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters: 571678236b844d108b1227119d8e90d1
I20240501 02:12:32.586007 1345 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 02:12:32.586001 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters: 571678236b844d108b1227119d8e90d1
I20240501 02:12:32.586125 1345 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 571678236b844d108b1227119d8e90d1)
I20240501 02:12:32.586242 1345 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:12:32.586465 1418 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 02:12:32.586612 1418 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 571678236b844d108b1227119d8e90d1)
I20240501 02:12:32.586750 1418 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:12:32.589298 1345 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.589478 1418 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:32.590355 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51"
I20240501 02:12:32.590651 1132 raft_consensus.cc:2379] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ff86a954ed44f79599a93fce287acc in current term 2: Already voted for candidate 610de482e0714825868dd5b03c7e1f51 in this term.
I20240501 02:12:32.590605 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "610de482e0714825868dd5b03c7e1f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30ff86a954ed44f79599a93fce287acc"
I20240501 02:12:32.591120 1001 raft_consensus.cc:2379] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 610de482e0714825868dd5b03c7e1f51 in current term 2: Already voted for candidate 30ff86a954ed44f79599a93fce287acc in this term.
I20240501 02:12:32.591766 1345 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 election: Requested vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:32.591922 935 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:32.592069 935 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc; no voters: 571678236b844d108b1227119d8e90d1, 610de482e0714825868dd5b03c7e1f51
I20240501 02:12:32.592111 1418 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: Requested vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:32.592404 1345 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 02:12:32.592814 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:32.592945 1066 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:32.593190 1418 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 02:12:32.600505 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.654412 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.681337 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.739641 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.770987 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.838600 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:32.869796 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:32.937474 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:32.965282 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:32.971884 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.049187 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.086783 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.172968 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.213424 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.302050 961 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.302892 960 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.307127 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.320916 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:33.328759 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.336462 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.347734 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.353096 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.366647 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.407455 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.423537 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.442762 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.454421 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:33.471958 1352 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:33.474606 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.490244 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.521157 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.540758 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.590190 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.594511 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.614091 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.644856 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.668658 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.695017 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.752594 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.757604 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.788755 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.807232 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.855849 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.890592 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.920527 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:33.964545 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:33.976701 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:34.000901 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:34.081260 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:34.091717 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
W20240501 02:12:34.123301 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:34.153698 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:34.162822 935 consensus_peers.cc:485] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 02:12:34.207450 1429 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:34.207653 1429 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:34.208333 1429 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:34.208904 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51" is_pre_election: true
I20240501 02:12:34.209146 1132 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 2.
I20240501 02:12:34.209724 937 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters:
W20240501 02:12:34.209810 959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54936: Illegal state: replica 30ff86a954ed44f79599a93fce287acc is not leader of this config: current role FOLLOWER
I20240501 02:12:34.210126 1429 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 02:12:34.210256 1429 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 02:12:34.210281 935 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:34.210362 1429 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 2 FOLLOWER]: Advancing to term 3
I20240501 02:12:34.212379 1429 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:34.213344 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340ca4359bc4d6c9f8630426f391f70" candidate_uuid: "30ff86a954ed44f79599a93fce287acc" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "610de482e0714825868dd5b03c7e1f51"
I20240501 02:12:34.213546 1132 raft_consensus.cc:3037] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 2 FOLLOWER]: Advancing to term 3
W20240501 02:12:34.214382 935 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:34.214632 1429 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 election: Requested vote from peers 610de482e0714825868dd5b03c7e1f51 (127.0.204.66:42855), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
I20240501 02:12:34.215934 1132 raft_consensus.cc:2454] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ff86a954ed44f79599a93fce287acc in term 3.
I20240501 02:12:34.216375 937 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ff86a954ed44f79599a93fce287acc, 610de482e0714825868dd5b03c7e1f51; no voters: 571678236b844d108b1227119d8e90d1
I20240501 02:12:34.216708 1429 raft_consensus.cc:2781] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 02:12:34.217142 1429 raft_consensus.cc:686] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [term 3 LEADER]: Becoming Leader. State: Replica: 30ff86a954ed44f79599a93fce287acc, State: Running, Role: LEADER
I20240501 02:12:34.217475 1429 consensus_queue.cc:237] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:34.220569 860 catalog_manager.cc:5484] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc reported cstate change: term changed from 1 to 3, leader changed from 571678236b844d108b1227119d8e90d1 (127.0.204.67) to 30ff86a954ed44f79599a93fce287acc (127.0.204.65). New cstate: current_term: 3 leader_uuid: "30ff86a954ed44f79599a93fce287acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } health_report { overall_health: UNKNOWN } } }
W20240501 02:12:34.253430 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:34.275353 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:34.298748 1132 raft_consensus.cc:1261] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Refusing update from remote peer 30ff86a954ed44f79599a93fce287acc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 02:12:34.299398 1429 consensus_queue.cc:1035] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 02:12:34.299854 935 consensus_peers.cc:485] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 02:12:34.723862 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:34.736411 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 02:12:35.445880 1352 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:36.714519 935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 02:12:36.716449 935 consensus_peers.cc:485] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 02:12:36.901134 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:37.073402 935 consensus_peers.cc:485] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 02:12:37.186981 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 02:12:37.547879 1352 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:37.773891 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240501 02:12:38.766528 1178 maintenance_manager.cc:419] P 610de482e0714825868dd5b03c7e1f51: Scheduling FlushMRSOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): perf score=1.000000
W20240501 02:12:38.801290 935 consensus_peers.cc:485] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 02:12:38.819275 1047 maintenance_manager.cc:419] P 30ff86a954ed44f79599a93fce287acc: Scheduling FlushMRSOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): perf score=1.000000
W20240501 02:12:39.438760 935 consensus_peers.cc:485] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:12:39.705601 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 02:12:39.935178 1069 maintenance_manager.cc:643] P 610de482e0714825868dd5b03c7e1f51: FlushMRSOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 1.168s user 0.518s sys 0.140s Metrics: {"bytes_written":3388,"cfile_init":1,"compiler_manager_pool.queue_time_us":3252,"delete_count":0,"dirs.queue_time_us":35786,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":2510,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2408,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7831,"thread_start_us":38514,"threads_started":2,"update_count":499}
I20240501 02:12:39.937348 1178 maintenance_manager.cc:419] P 610de482e0714825868dd5b03c7e1f51: Scheduling UndoDeltaBlockGCOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): 134955 bytes on disk
I20240501 02:12:39.940073 1069 maintenance_manager.cc:643] P 610de482e0714825868dd5b03c7e1f51: UndoDeltaBlockGCOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 02:12:39.941573 1178 maintenance_manager.cc:419] P 610de482e0714825868dd5b03c7e1f51: Scheduling MajorDeltaCompactionOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): perf score=0.027325
I20240501 02:12:40.045706 1069 maintenance_manager.cc:643] P 610de482e0714825868dd5b03c7e1f51: MajorDeltaCompactionOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 0.104s user 0.050s sys 0.019s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4396,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":656,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":1292,"lbm_read_time_us":436,"lbm_reads_lt_1ms":20,"lbm_write_time_us":334,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":3617,"threads_started":1,"update_count":499}
I20240501 02:12:40.341929 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:12:40.350208 938 maintenance_manager.cc:643] P 30ff86a954ed44f79599a93fce287acc: FlushMRSOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 1.530s user 0.584s sys 0.058s Metrics: {"bytes_written":4000,"cfile_init":1,"compiler_manager_pool.queue_time_us":196010,"compiler_manager_pool.run_cpu_time_us":680447,"compiler_manager_pool.run_wall_time_us":1462413,"delete_count":0,"dirs.queue_time_us":1890,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1140,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2434,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7927,"thread_start_us":13996,"threads_started":2,"update_count":595}
I20240501 02:12:40.351670 1047 maintenance_manager.cc:419] P 30ff86a954ed44f79599a93fce287acc: Scheduling MajorDeltaCompactionOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): perf score=0.031860
I20240501 02:12:40.399173 938 maintenance_manager.cc:643] P 30ff86a954ed44f79599a93fce287acc: MajorDeltaCompactionOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 0.047s user 0.036s sys 0.011s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5066,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":533,"lbm_read_time_us":284,"lbm_reads_lt_1ms":24,"lbm_write_time_us":342,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":294,"threads_started":1,"update_count":595}
I20240501 02:12:40.399981 1047 maintenance_manager.cc:419] P 30ff86a954ed44f79599a93fce287acc: Scheduling UndoDeltaBlockGCOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f): 135975 bytes on disk
I20240501 02:12:40.400595 938 maintenance_manager.cc:643] P 30ff86a954ed44f79599a93fce287acc: UndoDeltaBlockGCOp(6e9fb5b0cc2a40b58c067ccbc8b28b8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 02:12:40.927345 1352 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:12:41.002357 817 linked_list-test-util.h:417] Sample at HT timestamp: 7022713081794392064 Inserted count: 15750
W20240501 02:12:41.312312 935 consensus_peers.cc:485] T d3c31b5c984043f38cd869f5b21c067e P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 02:12:41.788769 935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 02:12:41.790124 935 consensus_peers.cc:485] T 2340ca4359bc4d6c9f8630426f391f70 P 30ff86a954ed44f79599a93fce287acc -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 02:12:42.203320 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 02:12:42.903378 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 02:12:42.942608 1352 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f0c1136edcd google::LogMessage::Fail() at ??:0
@ 0x7f0c11372b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0c1136e7cc google::LogMessage::Flush() at ??:0
@ 0x7f0c1136ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c13f97c80 kudu::operator-() at ??:0
@ 0x7f0c20c1814a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f0c20be25de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f0c20be2336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f0c20b8f2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f0c2365cd87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f0c2366b93c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f0c2366b7a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0c1d0a102a std::function<>::operator()() at ??:0
@ 0x7f0c1d0997e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f0c1d0947f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f0c1d092927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f0c1d0d0089 handle_request at ??:0
@ 0x7f0c1d0d276b worker_thread at ??:0
@ 0x7f0c173e36db start_thread at ??:0
@ 0x7f0c1022171f clone at ??:0
W20240501 02:12:43.234252 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
W20240501 02:12:43.236717 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 02:12:43.240521 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.263350 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.292209 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.332149 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.385668 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.445648 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.512966 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.518714 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:12:43.590829 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.604501 1090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.604529 1091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.623975 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.654208 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.660785 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.672694 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.695170 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.728300 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.763892 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.784062 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.833614 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.860244 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.910924 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:43.963869 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.024150 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.054456 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.122815 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.227809 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.236471 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.263661 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:44.331107 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.338809 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.376943 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:44.388384 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:12:44.434466 1525 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:44.434664 1525 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:44.435356 1525 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:44.437034 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:44.437250 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:12:44.437373 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:44.437613 1525 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 02:12:44.450059 1525 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:44.450263 1525 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:44.450973 1525 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:44.452512 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:44.452723 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:44.452872 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:44.453171 1525 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:12:44.460824 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.530138 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.591848 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.685111 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.694025 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.726998 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 02:12:44.802013 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.875015 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:44.977463 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.031121 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.142735 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.145902 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.227521 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.291636 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:45.317096 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.427320 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:45.492219 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:45.493105 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.619891 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.624119 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.703542 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.767357 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 02:12:45.813602 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:45.918926 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:46.001017 1525 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:46.001263 1525 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:46.002178 1525 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:46.003799 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:46.005234 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:46.005386 1066 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:46.005666 1525 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:12:46.017452 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.140906 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.231499 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.295924 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:46.373759 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.449537 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:46.513229 1525 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:46.513588 1525 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:46.515638 1525 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:46.516878 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:46.517340 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:46.517449 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:46.517760 1525 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:12:46.579546 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:46.608289 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.700017 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.843731 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:46.950580 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.088236 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.158494 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 02:12:47.217706 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.308272 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:47.345225 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.497994 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.607268 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:47.657332 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:47.781754 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:47.882182 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:48.068701 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:48.256098 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:48.352679 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:48.372285 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:48.478950 1067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 02:12:48.480541 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:12:48.551190 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:48.684139 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:48.692584 1525 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:48.692793 1525 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:48.693557 1525 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:48.695006 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:48.695667 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:48.695820 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:48.696103 1525 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:12:48.734288 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:48.854575 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:49.003093 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:49.169487 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:49.206959 1525 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:49.207163 1525 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:49.207975 1525 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:49.209770 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:49.210450 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:12:49.210564 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:49.210831 1525 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:12:49.331666 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:49.379073 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:49.489523 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:49.670256 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:49.822041 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:49.834362 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:49.896777 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 02:12:50.015110 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:50.160317 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:50.371248 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:50.386507 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:50.507161 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:50.736853 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:50.788576 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 02:12:50.859995 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:50.902806 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:51.108801 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:51.109758 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:51.364005 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:51.477947 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:51.610303 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:51.752251 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:51.992497 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:52.002969 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:52.154780 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:52.384315 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:52.473587 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 02:12:52.509580 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:52.561841 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:52.792932 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:52.976593 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:53.102924 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:53.202335 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:53.280112 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 02:12:53.397423 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:53.483129 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 02:12:53.539240 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:53.623005 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:53.689404 1536 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:53.689643 1536 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:53.690361 1536 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:53.692194 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:53.692348 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:53.692520 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:53.692832 1536 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:12:53.831815 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:53.889134 1536 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:53.889397 1536 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:53.890159 1536 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:53.891541 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:53.891911 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:12:53.892072 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:53.892439 1536 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:12:54.055065 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:54.156333 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:54.271612 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:54.491652 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:54.571653 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:54.722432 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:54.937263 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:55.169834 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 02:12:55.180577 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:55.233572 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:55.293753 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 02:12:55.396057 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:55.643715 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:55.681499 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:55.863083 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:56.120683 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:56.297724 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:56.337920 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:56.567309 1536 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:56.567638 1536 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:56.568691 1536 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:56.570696 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:56.571436 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:12:56.571703 1067 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:56.572321 1536 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:12:56.603114 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:56.769102 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:56.820794 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:57.102113 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:57.105476 1536 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:57.105757 1536 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:57.106760 1536 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:57.108282 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:57.108518 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:12:57.108697 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:57.109027 1536 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:12:57.311527 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:57.362264 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:57.546514 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 02:12:57.603667 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:57.774004 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:12:57.809283 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:58.072739 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 02:12:58.117056 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:58.319356 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:58.445044 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:58.538969 1067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 02:12:58.640604 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:58.690819 1536 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:12:58.691121 1536 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:12:58.691867 1536 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:12:58.693655 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:58.693837 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:12:58.694027 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:12:58.694391 1536 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:12:58.814316 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:58.837771 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:59.176043 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:59.366694 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:12:59.559473 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:12:59.714972 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:12:59.837443 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 02:12:59.871162 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:12:59.904796 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:00.268664 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:00.370362 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 02:13:00.450305 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:00.648964 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:00.821532 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:00.971335 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:01.012557 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:01.379693 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:01.586771 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:01.743130 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:01.947135 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:01.975925 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:02.171196 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:02.332965 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 02:13:02.524734 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:02.761075 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:02.843618 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 02:13:02.879118 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:03.087742 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:03.111961 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:03.355825 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:03.655922 1067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 02:13:03.708222 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:03.960181 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:03.991432 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:04.113790 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:04.167420 1536 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:04.167944 1536 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:04.168956 1536 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:04.171058 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:04.171644 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:04.171830 1066 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:04.172209 1536 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:13:04.319778 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:04.576175 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:04.739838 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 02:13:04.939725 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:04.958768 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 02:13:05.148532 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:05.164580 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:05.199168 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:05.570297 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:05.837081 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:06.160231 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:06.205718 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:06.268577 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:06.483096 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:06.851434 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:07.139168 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:07.215111 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 02:13:07.238621 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:07.394682 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:07.504796 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:07.612740 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 02:13:07.802894 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:08.168946 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:08.250552 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:08.473572 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:08.557783 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:08.840185 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:08.900113 1067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 02:13:09.152490 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:09.298398 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:09.527467 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:09.623726 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 02:13:09.685335 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:09.844079 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:09.929366 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 02:13:10.211766 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:10.306026 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:10.540313 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:10.832266 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:10.910080 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:11.079739 1536 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:11.080053 1536 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:11.081135 1536 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:11.083456 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:11.083650 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:11.083743 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:11.084053 1536 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:13:11.249923 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:11.319283 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:11.617983 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:11.954308 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:11.969765 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:12.111160 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 02:13:12.289409 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 02:13:12.328485 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:12.450390 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:12.691145 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:13.054095 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:13.058486 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:13.423727 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:13.478487 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:13.785859 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:13.914203 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 02:13:14.157601 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:14.162173 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:14.528311 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:14.625818 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 02:13:14.664599 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:14.865944 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 02:13:14.916138 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:15.232122 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:15.271608 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:15.678660 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:15.709874 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:16.030494 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:16.248853 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:16.441948 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:16.735204 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:16.799023 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:17.218967 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:17.272904 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 02:13:17.340087 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:17.347954 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 02:13:17.574296 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:17.740432 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:17.942795 1548 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:17.943014 1548 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:17.943686 1548 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:17.944967 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:17.945379 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:17.945523 1066 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:17.945816 1548 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:13:18.002887 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:18.358928 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:18.408015 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:18.793427 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:18.822499 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:18.953269 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 02:13:19.155359 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:19.461534 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:19.599299 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:19.750375 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 02:13:19.825366 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 02:13:19.868443 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:19.959319 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:20.416307 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:20.524876 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:20.774024 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:20.964828 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:21.239290 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:21.601195 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:21.623852 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:22.068109 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:22.080296 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:22.277148 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 02:13:22.431171 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:22.517593 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 02:13:22.738145 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:22.910980 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:23.190676 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:23.270166 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:23.760407 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:23.846711 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:24.122637 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:24.210234 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 02:13:24.335137 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:24.623757 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:24.746820 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 02:13:24.768803 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 02:13:24.918933 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:24.980393 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:25.483485 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:25.492604 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:25.843784 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:26.024510 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:26.367610 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:26.644428 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:26.724623 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:27.134452 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:27.258836 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:27.363343 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 02:13:27.396001 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 02:13:27.617619 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:27.827327 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:28.153074 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:28.261102 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:28.511421 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:29.025031 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:29.061129 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:29.192850 1560 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:29.193042 1560 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:29.193784 1560 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:29.195655 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:29.195773 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:29.195919 1067 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:29.196210 1560 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:13:29.363011 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:29.415401 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:29.557125 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 02:13:29.973455 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:30.027918 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:30.134306 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 02:13:30.151417 1564 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:30.151604 1564 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:30.152741 1564 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:30.154318 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:30.154592 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:30.154728 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:30.155052 1564 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:13:30.158202 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 02:13:30.333065 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:30.419267 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:30.892242 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:31.167246 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:31.253942 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:31.502112 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:13:31.605727 1566 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:31.605973 1566 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:31.606760 1566 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:31.608476 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:31.608707 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:31.608842 1067 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:31.609109 1566 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:13:31.823506 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:32.186985 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:32.416822 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:32.645540 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:32.728770 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 02:13:32.758687 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:32.826934 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 02:13:33.128625 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:33.444825 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:33.710685 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:33.731010 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:13:33.940474 1570 raft_consensus.cc:482] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:33.940776 1570 raft_consensus.cc:504] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:33.942018 1570 leader_election.cc:288] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:33.944695 1066 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:33.944869 1067 leader_election.cc:334] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:33.945037 1067 leader_election.cc:302] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:33.945416 1570 raft_consensus.cc:2726] T 2340ca4359bc4d6c9f8630426f391f70 P 610de482e0714825868dd5b03c7e1f51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 02:13:34.079882 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:34.628613 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:34.663278 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:34.734371 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20240501 02:13:34.815677 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:35.038640 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:35.169500 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 02:13:35.232110 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 02:13:35.632704 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:35.683437 1312 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
I20240501 02:13:35.951424 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:36.004033 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:36.608729 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:36.893431 1312 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:36.978821 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:37.057337 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:37.559038 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 02:13:37.593683 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:37.594897 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 02:13:37.923777 1311 meta_cache.cc:302] tablet 2340ca4359bc4d6c9f8630426f391f70: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:37.963954 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:38.102463 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:38.589562 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:38.961880 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
I20240501 02:13:39.167272 1570 raft_consensus.cc:482] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30ff86a954ed44f79599a93fce287acc)
I20240501 02:13:39.167482 1570 raft_consensus.cc:504] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ff86a954ed44f79599a93fce287acc" member_type: VOTER last_known_addr { host: "127.0.204.65" port: 35425 } } peers { permanent_uuid: "610de482e0714825868dd5b03c7e1f51" member_type: VOTER last_known_addr { host: "127.0.204.66" port: 42855 } } peers { permanent_uuid: "571678236b844d108b1227119d8e90d1" member_type: VOTER last_known_addr { host: "127.0.204.67" port: 43585 } }
I20240501 02:13:39.168195 1570 leader_election.cc:288] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425), 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585)
W20240501 02:13:39.169603 1067 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111)
W20240501 02:13:39.169835 1066 leader_election.cc:334] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:39.169929 1066 leader_election.cc:302] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 610de482e0714825868dd5b03c7e1f51; no voters: 30ff86a954ed44f79599a93fce287acc, 571678236b844d108b1227119d8e90d1
I20240501 02:13:39.170197 1570 raft_consensus.cc:2726] T d3c31b5c984043f38cd869f5b21c067e P 610de482e0714825868dd5b03c7e1f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:13:39.184466 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:39.204532 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
W20240501 02:13:39.592938 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:39.895998 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 02:13:39.897131 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 02:13:39.964213 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:40.055768 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 02:13:40.210942 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:40.236191 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:40.601847 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:40.975493 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:41.227478 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:41.307708 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:41.617058 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:41.997678 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:42.145826 1066 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585): Couldn't send request to peer 571678236b844d108b1227119d8e90d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 02:13:42.255537 1311 meta_cache.cc:302] tablet d3c31b5c984043f38cd869f5b21c067e: replica 571678236b844d108b1227119d8e90d1 (127.0.204.67:43585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.204.67:43585: connect: Connection refused (error 111)
I20240501 02:13:42.386430 1394 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:13:42.636400 1067 consensus_peers.cc:485] T 6e9fb5b0cc2a40b58c067ccbc8b28b8f P 610de482e0714825868dd5b03c7e1f51 -> Peer 30ff86a954ed44f79599a93fce287acc (127.0.204.65:35425): Couldn't send request to peer 30ff86a954ed44f79599a93fce287acc. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.65:35425: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 02:13:42.646049 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
W20240501 02:13:42.991241 1313 batcher.cc:441] Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.992115 817 client-test-util.cc:49] 9 failed ops. First 10 errors follow
I20240501 02:13:42.992316 817 client-test-util.cc:55] Op INSERT int64 rand_key=5938602475919835140, int64 link_to=-2392534088480391164, int64 insert_ts=1714529562937916 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.992641 817 client-test-util.cc:55] Op INSERT int64 rand_key=5902012196608213001, int64 link_to=9059295571765821449, int64 insert_ts=1714529562938206 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.992739 817 client-test-util.cc:55] Op INSERT int64 rand_key=5865703392273301518, int64 link_to=2064099683325771790, int64 insert_ts=1714529562938491 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.992856 817 client-test-util.cc:55] Op INSERT int64 rand_key=5829113112961679379, int64 link_to=-4930814730137567213, int64 insert_ts=1714529562938755 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.992959 817 client-test-util.cc:55] Op INSERT int64 rand_key=3651698061256163348, int64 link_to=-4193870930497241068, int64 insert_ts=1714529562938796 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.993064 817 client-test-util.cc:55] Op INSERT int64 rand_key=3615107781944541209, int64 link_to=7257677254772260889, int64 insert_ts=1714529562939045 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.993376 817 client-test-util.cc:55] Op INSERT int64 rand_key=3578517502632919070, int64 link_to=262622103820632094, int64 insert_ts=1714529562939285 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.993505 817 client-test-util.cc:55] Op INSERT int64 rand_key=4297293499008352299, int64 link_to=4259862262282780715, int64 insert_ts=1714529562940027 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
I20240501 02:13:42.993604 817 client-test-util.cc:55] Op INSERT int64 rand_key=4260703219696730160, int64 link_to=-2735052151180558288, int64 insert_ts=1714529562940293 had status Timed out: Failed to write batch of 9 ops to tablet 2340ca4359bc4d6c9f8630426f391f70 after 341 attempt(s): Failed to write to server: (no server available): Write(tablet: 2340ca4359bc4d6c9f8630426f391f70, num_ops: 9, num_attempts: 341) passed its deadline: Network error: recv got EOF from 127.0.204.65:35425 (error 108)
F20240501 02:13:42.993712 817 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714529622 (unix time) try "date -d @1714529622" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000331) received by PID 817 (TID 0x7f5c64844d80) from PID 817; stack trace: ***
@ 0x7f5c6c251980 (unknown) at ??:0
@ 0x7f5c652fcfb7 gsignal at ??:0
@ 0x7f5c652fe921 abort at ??:0
@ 0x7f5c68025dcd google::LogMessage::Fail() at ??:0
@ 0x7f5c68029b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5c680257cc google::LogMessage::Flush() at ??:0
@ 0x7f5c68026f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5c77abbefd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed8c2 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test-util.h:?
@ 0x3d6588 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8889 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7f5c69ec00ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
W20240501 02:13:43.024487 1092 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38738: Illegal state: replica 610de482e0714825868dd5b03c7e1f51 is not leader of this config: current role FOLLOWER
@ 0x7f5c69eb4bdb testing::Test::Run() at ??:0
@ 0x7f5c69eb4d9d testing::TestInfo::Run() at ??:0
@ 0x7f5c69eb5377 testing::TestSuite::Run() at ??:0
@ 0x7f5c69eb577c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f5c69ec060d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f5c69eb4e63 testing::UnitTest::Run() at ??:0
@ 0x7f5c7eb9f3c2 RUN_ALL_TESTS() at ??:0
@ 0x7f5c7eb9d8d9 main at ??:0
@ 0x7f5c652dfbf7 __libc_start_main at ??:0
@ 0x2ebf5a _start at ??:?