[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 09:30:46.745908 396 test_util.cc:268] Using random seed: 106561306
I20240430 09:30:46.752578 396 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 09:30:46.752633 396 linked_list-test.cc:101] --------------
I20240430 09:30:46.752651 396 linked_list-test.cc:102] 50 chains
I20240430 09:30:46.752681 396 linked_list-test.cc:103] 3 tablets
I20240430 09:30:46.752712 396 linked_list-test.cc:104] Mutations on
I20240430 09:30:46.752732 396 linked_list-test.cc:105] --------------
I20240430 09:30:46.752808 396 ts_itest-base.cc:112] Starting cluster with:
I20240430 09:30:46.752840 396 ts_itest-base.cc:113] --------------
I20240430 09:30:46.752859 396 ts_itest-base.cc:114] 3 tablet servers
I20240430 09:30:46.752888 396 ts_itest-base.cc:115] 3 replicas per TS
I20240430 09:30:46.752923 396 ts_itest-base.cc:116] --------------
2024-04-30T09:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T09:30:46Z Disabled control of system clock
I20240430 09:30:46.762627 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXye1oe/build/debug/bin/kudu
/tmp/dist-test-taskXye1oe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.62:39647
--webserver_interface=127.0.99.62
--webserver_port=0
--builtin_ntp_servers=127.0.99.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.62:39647
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 09:30:46.867347 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:30:46.867578 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:30:46.867640 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:30:46.870582 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 09:30:46.870631 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:30:46.870651 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 09:30:46.870666 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 09:30:46.874157 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.99.62:39647
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.62:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 09:12:56 UTC on 1a15035a6830
build id 49
I20240430 09:30:46.874972 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:30:46.876109 408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:30:46.883968 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:46.884675 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:46.885497 408 server_base.cc:1034] running on GCE node
W20240430 09:30:46.886176 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:46.886709 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:30:46.887624 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:30:46.888770 408 hybrid_clock.cc:648] HybridClock initialized: now 1714469446888724 us; error 61 us; skew 500 ppm
I20240430 09:30:46.890638 408 webserver.cc:460] Webserver started at http://127.0.99.62:45235/ using document root <none> and password file <none>
I20240430 09:30:46.891201 408 fs_manager.cc:359] Metadata directory not provided
I20240430 09:30:46.891256 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:30:46.891418 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:30:46.893024 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data/instance:
uuid: "3b065f000a14498882454b74ce8a696e"
format_stamp: "Formatted at 2024-04-30 09:30:46 on dist-test-slave-6lsn"
I20240430 09:30:46.893467 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal/instance:
uuid: "3b065f000a14498882454b74ce8a696e"
format_stamp: "Formatted at 2024-04-30 09:30:46 on dist-test-slave-6lsn"
I20240430 09:30:46.896786 408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240430 09:30:46.898928 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:46.899930 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240430 09:30:46.900041 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal
uuid: "3b065f000a14498882454b74ce8a696e"
format_stamp: "Formatted at 2024-04-30 09:30:46 on dist-test-slave-6lsn"
I20240430 09:30:46.900135 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:30:46.918330 408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 09:30:46.919004 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:30:46.919188 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:30:46.925629 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:39647
I20240430 09:30:46.925638 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:39647 every 8 connection(s)
I20240430 09:30:46.926638 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/data/info.pb
I20240430 09:30:46.928105 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXye1oe/build/debug/bin/kudu as pid 408
I20240430 09:30:46.928253 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/master-0/wal/instance
I20240430 09:30:46.929694 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:46.935412 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: Bootstrap starting.
I20240430 09:30:46.938026 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:46.938853 475 log.cc:824] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: Log is configured to *not* fsync() on all Append() calls
I20240430 09:30:46.940783 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: No bootstrap required, opened a new log
I20240430 09:30:46.943629 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } }
I20240430 09:30:46.943809 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:46.943886 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b065f000a14498882454b74ce8a696e, State: Initialized, Role: FOLLOWER
I20240430 09:30:46.944317 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } }
I20240430 09:30:46.944437 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 09:30:46.944496 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 09:30:46.944602 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:46.945374 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } }
I20240430 09:30:46.945694 475 leader_election.cc:302] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b065f000a14498882454b74ce8a696e; no voters:
I20240430 09:30:46.945997 475 leader_election.cc:288] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 09:30:46.946127 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:30:46.946333 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [term 1 LEADER]: Becoming Leader. State: Replica: 3b065f000a14498882454b74ce8a696e, State: Running, Role: LEADER
I20240430 09:30:46.947036 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [sys.catalog]: configured and running, proceeding with master startup.
I20240430 09:30:46.946681 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } }
I20240430 09:30:46.948523 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b065f000a14498882454b74ce8a696e. Latest consensus state: current_term: 1 leader_uuid: "3b065f000a14498882454b74ce8a696e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } } }
I20240430 09:30:46.948645 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [sys.catalog]: This master's current role is: LEADER
I20240430 09:30:46.948956 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b065f000a14498882454b74ce8a696e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b065f000a14498882454b74ce8a696e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39647 } } }
I20240430 09:30:46.949040 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e [sys.catalog]: This master's current role is: LEADER
W20240430 09:30:46.951704 495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 09:30:46.951774 495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 09:30:46.951838 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 09:30:46.952595 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 09:30:46.958020 492 catalog_manager.cc:1347] Generated new cluster ID: 15e4a39c9dfb46eab485c837454c4adc
I20240430 09:30:46.958091 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 09:30:46.986651 492 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 09:30:46.987596 492 catalog_manager.cc:1504] Loading token signing keys...
I20240430 09:30:46.994920 492 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3b065f000a14498882454b74ce8a696e: Generated new TSK 0
I20240430 09:30:46.995595 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 09:30:47.000001 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXye1oe/build/debug/bin/kudu
/tmp/dist-test-taskXye1oe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.1:0
--local_ip_for_outbound_sockets=127.0.99.1
--webserver_interface=127.0.99.1
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39647
--builtin_ntp_servers=127.0.99.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 09:30:47.107780 499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:30:47.107991 499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:30:47.108063 499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:30:47.111174 499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:30:47.111251 499 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:30:47.111289 499 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:30:47.111330 499 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:30:47.111404 499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.1
I20240430 09:30:47.115073 499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39647
--never_fsync=true
--heap_profile_path=/tmp/kudu.499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.1
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 09:12:56 UTC on 1a15035a6830
build id 49
I20240430 09:30:47.115962 499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:30:47.117060 499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:30:47.125075 507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:47.125854 504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:47.126691 499 server_base.cc:1034] running on GCE node
W20240430 09:30:47.127359 505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:47.127722 499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:30:47.128319 499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:30:47.129487 499 hybrid_clock.cc:648] HybridClock initialized: now 1714469447129476 us; error 28 us; skew 500 ppm
I20240430 09:30:47.131450 499 webserver.cc:460] Webserver started at http://127.0.99.1:45933/ using document root <none> and password file <none>
I20240430 09:30:47.132001 499 fs_manager.cc:359] Metadata directory not provided
I20240430 09:30:47.132063 499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:30:47.132233 499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:30:47.133855 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data/instance:
uuid: "960dc67926f24ed78bf460935ffd13cd"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.134382 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal/instance:
uuid: "960dc67926f24ed78bf460935ffd13cd"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.137537 499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240430 09:30:47.139632 513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.140571 499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240430 09:30:47.140673 499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal
uuid: "960dc67926f24ed78bf460935ffd13cd"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.140756 499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:30:47.158097 499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 09:30:47.158733 499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:30:47.158895 499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:30:47.159449 499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:30:47.160377 499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:30:47.160427 499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.160486 499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:30:47.160518 499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.169495 499 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:44385
I20240430 09:30:47.169528 626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:44385 every 8 connection(s)
I20240430 09:30:47.170495 499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/data/info.pb
I20240430 09:30:47.175550 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXye1oe/build/debug/bin/kudu as pid 499
I20240430 09:30:47.175701 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-0/wal/instance
I20240430 09:30:47.177842 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXye1oe/build/debug/bin/kudu
/tmp/dist-test-taskXye1oe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.2:0
--local_ip_for_outbound_sockets=127.0.99.2
--webserver_interface=127.0.99.2
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39647
--builtin_ntp_servers=127.0.99.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 09:30:47.179592 627 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39647
I20240430 09:30:47.179824 627 heartbeater.cc:464] Registering TS with master...
I20240430 09:30:47.180338 627 heartbeater.cc:511] Master 127.0.99.62:39647 requested a full tablet report, sending...
I20240430 09:30:47.181702 439 ts_manager.cc:194] Registered new tserver with Master: 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385)
I20240430 09:30:47.182848 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.1:34717
W20240430 09:30:47.283064 630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:30:47.283277 630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:30:47.283340 630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:30:47.286422 630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:30:47.286500 630 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:30:47.286516 630 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:30:47.286547 630 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:30:47.286620 630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.2
I20240430 09:30:47.290089 630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39647
--never_fsync=true
--heap_profile_path=/tmp/kudu.630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.2
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 09:12:56 UTC on 1a15035a6830
build id 49
I20240430 09:30:47.290947 630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:30:47.292055 630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:30:47.300531 635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:47.301306 636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:47.301925 638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:47.302785 630 server_base.cc:1034] running on GCE node
I20240430 09:30:47.303232 630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:30:47.303894 630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:30:47.305049 630 hybrid_clock.cc:648] HybridClock initialized: now 1714469447305046 us; error 45 us; skew 500 ppm
I20240430 09:30:47.306932 630 webserver.cc:460] Webserver started at http://127.0.99.2:41881/ using document root <none> and password file <none>
I20240430 09:30:47.307504 630 fs_manager.cc:359] Metadata directory not provided
I20240430 09:30:47.307566 630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:30:47.307758 630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:30:47.309886 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data/instance:
uuid: "ac0d750046af4e748b72bfba40f38fbe"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.310407 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal/instance:
uuid: "ac0d750046af4e748b72bfba40f38fbe"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.313724 630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240430 09:30:47.316042 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.317075 630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240430 09:30:47.317198 630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal
uuid: "ac0d750046af4e748b72bfba40f38fbe"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.317286 630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:30:47.330988 630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 09:30:47.331641 630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:30:47.331800 630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:30:47.332315 630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:30:47.333245 630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:30:47.333295 630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.333354 630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:30:47.333385 630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.342970 630 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:33585
I20240430 09:30:47.343025 757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:33585 every 8 connection(s)
I20240430 09:30:47.343978 630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/data/info.pb
I20240430 09:30:47.353147 758 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39647
I20240430 09:30:47.353327 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXye1oe/build/debug/bin/kudu as pid 630
I20240430 09:30:47.353375 758 heartbeater.cc:464] Registering TS with master...
I20240430 09:30:47.353461 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-1/wal/instance
I20240430 09:30:47.354013 758 heartbeater.cc:511] Master 127.0.99.62:39647 requested a full tablet report, sending...
I20240430 09:30:47.355118 439 ts_manager.cc:194] Registered new tserver with Master: ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
I20240430 09:30:47.355643 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXye1oe/build/debug/bin/kudu
/tmp/dist-test-taskXye1oe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.3:0
--local_ip_for_outbound_sockets=127.0.99.3
--webserver_interface=127.0.99.3
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39647
--builtin_ntp_servers=127.0.99.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 09:30:47.355665 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.2:34445
W20240430 09:30:47.464668 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:30:47.464879 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:30:47.464959 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:30:47.467969 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:30:47.468042 761 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:30:47.468060 761 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:30:47.468079 761 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:30:47.468120 761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.3
I20240430 09:30:47.471660 761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39647
--never_fsync=true
--heap_profile_path=/tmp/kudu.761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.3
--log_dir=/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 09:12:56 UTC on 1a15035a6830
build id 49
I20240430 09:30:47.472523 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:30:47.473631 761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:30:47.481535 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:47.482338 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:30:47.483047 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:30:47.483873 761 server_base.cc:1034] running on GCE node
I20240430 09:30:47.484242 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:30:47.484757 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:30:47.485913 761 hybrid_clock.cc:648] HybridClock initialized: now 1714469447485899 us; error 28 us; skew 500 ppm
I20240430 09:30:47.487746 761 webserver.cc:460] Webserver started at http://127.0.99.3:33375/ using document root <none> and password file <none>
I20240430 09:30:47.488386 761 fs_manager.cc:359] Metadata directory not provided
I20240430 09:30:47.488484 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:30:47.488737 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:30:47.490387 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data/instance:
uuid: "40b0730cd16f4499976c13c7a0f22e63"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.490826 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal/instance:
uuid: "40b0730cd16f4499976c13c7a0f22e63"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.494163 761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240430 09:30:47.496382 775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.497380 761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 09:30:47.497476 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal
uuid: "40b0730cd16f4499976c13c7a0f22e63"
format_stamp: "Formatted at 2024-04-30 09:30:47 on dist-test-slave-6lsn"
I20240430 09:30:47.497571 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:30:47.517544 761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 09:30:47.518216 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:30:47.518374 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:30:47.518899 761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:30:47.519759 761 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:30:47.519805 761 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.519850 761 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:30:47.519881 761 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 09:30:47.530444 761 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:42655
I20240430 09:30:47.530495 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:42655 every 8 connection(s)
I20240430 09:30:47.531740 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/data/info.pb
I20240430 09:30:47.540493 889 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39647
I20240430 09:30:47.540728 889 heartbeater.cc:464] Registering TS with master...
I20240430 09:30:47.541157 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXye1oe/build/debug/bin/kudu as pid 761
I20240430 09:30:47.541247 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXye1oe/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714469446735262-396-0/linked-list-cluster/ts-2/wal/instance
I20240430 09:30:47.541306 889 heartbeater.cc:511] Master 127.0.99.62:39647 requested a full tablet report, sending...
I20240430 09:30:47.542356 438 ts_manager.cc:194] Registered new tserver with Master: 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
I20240430 09:30:47.543062 438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.3:45113
I20240430 09:30:47.543692 396 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:30:47.554879 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56860:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240430 09:30:47.557614 439 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 09:30:47.573815 561 tablet_service.cc:1467] Processing CreateTablet for tablet 5b457e347ac7472291ba6d38c7cfb90c (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:30:47.573812 823 tablet_service.cc:1467] Processing CreateTablet for tablet 5b457e347ac7472291ba6d38c7cfb90c (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:30:47.574481 821 tablet_service.cc:1467] Processing CreateTablet for tablet 28fef7488e904e04a08ab7025b4e4409 (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:30:47.574757 822 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72c630f2044d14b15ef5d55ebec76f (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:30:47.575053 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b457e347ac7472291ba6d38c7cfb90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.575062 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fef7488e904e04a08ab7025b4e4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.575367 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72c630f2044d14b15ef5d55ebec76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.575151 692 tablet_service.cc:1467] Processing CreateTablet for tablet 5b457e347ac7472291ba6d38c7cfb90c (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:30:47.573813 559 tablet_service.cc:1467] Processing CreateTablet for tablet 28fef7488e904e04a08ab7025b4e4409 (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:30:47.576095 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b457e347ac7472291ba6d38c7cfb90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.576519 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b457e347ac7472291ba6d38c7cfb90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.574188 560 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72c630f2044d14b15ef5d55ebec76f (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:30:47.577513 690 tablet_service.cc:1467] Processing CreateTablet for tablet 28fef7488e904e04a08ab7025b4e4409 (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:30:47.577754 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fef7488e904e04a08ab7025b4e4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.578524 691 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72c630f2044d14b15ef5d55ebec76f (DEFAULT_TABLE table=TestTable [id=9dde9a130f6846a3a6df211ba4a94aea]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:30:47.578739 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72c630f2044d14b15ef5d55ebec76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.580055 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28fef7488e904e04a08ab7025b4e4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.580229 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72c630f2044d14b15ef5d55ebec76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:30:47.584996 908 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Bootstrap starting.
I20240430 09:30:47.585506 907 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Bootstrap starting.
I20240430 09:30:47.586241 906 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Bootstrap starting.
I20240430 09:30:47.586927 908 tablet_bootstrap.cc:654] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.587093 907 tablet_bootstrap.cc:654] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.587837 907 log.cc:824] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Log is configured to *not* fsync() on all Append() calls
I20240430 09:30:47.587893 908 log.cc:824] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Log is configured to *not* fsync() on all Append() calls
I20240430 09:30:47.588075 906 tablet_bootstrap.cc:654] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.588663 906 log.cc:824] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Log is configured to *not* fsync() on all Append() calls
I20240430 09:30:47.589774 907 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: No bootstrap required, opened a new log
I20240430 09:30:47.589928 907 ts_tablet_manager.cc:1367] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240430 09:30:47.590004 906 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: No bootstrap required, opened a new log
I20240430 09:30:47.590157 906 ts_tablet_manager.cc:1367] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240430 09:30:47.590998 908 tablet_bootstrap.cc:492] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: No bootstrap required, opened a new log
I20240430 09:30:47.591156 908 ts_tablet_manager.cc:1367] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240430 09:30:47.592485 907 raft_consensus.cc:348] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.592661 907 raft_consensus.cc:374] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.592733 907 raft_consensus.cc:729] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0d750046af4e748b72bfba40f38fbe, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.593154 907 consensus_queue.cc:260] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.593416 906 raft_consensus.cc:348] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.593624 906 raft_consensus.cc:374] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.593679 906 raft_consensus.cc:729] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b0730cd16f4499976c13c7a0f22e63, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.593843 758 heartbeater.cc:503] Master 127.0.99.62:39647 was elected leader, sending a full tablet report...
I20240430 09:30:47.594096 906 consensus_queue.cc:260] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.594166 908 raft_consensus.cc:348] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.594336 908 raft_consensus.cc:374] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.594381 908 raft_consensus.cc:729] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960dc67926f24ed78bf460935ffd13cd, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.594743 908 consensus_queue.cc:260] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.594995 906 ts_tablet_manager.cc:1398] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240430 09:30:47.595333 906 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63: Bootstrap starting.
I20240430 09:30:47.595342 889 heartbeater.cc:503] Master 127.0.99.62:39647 was elected leader, sending a full tablet report...
I20240430 09:30:47.595476 908 ts_tablet_manager.cc:1398] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240430 09:30:47.595772 908 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: Bootstrap starting.
I20240430 09:30:47.596297 627 heartbeater.cc:503] Master 127.0.99.62:39647 was elected leader, sending a full tablet report...
I20240430 09:30:47.596354 906 tablet_bootstrap.cc:654] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.596751 908 tablet_bootstrap.cc:654] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.597887 907 ts_tablet_manager.cc:1398] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240430 09:30:47.597976 906 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63: No bootstrap required, opened a new log
I20240430 09:30:47.598055 906 ts_tablet_manager.cc:1367] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240430 09:30:47.598456 908 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: No bootstrap required, opened a new log
I20240430 09:30:47.598526 908 ts_tablet_manager.cc:1367] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240430 09:30:47.598578 906 raft_consensus.cc:348] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.598666 906 raft_consensus.cc:374] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.598697 906 raft_consensus.cc:729] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b0730cd16f4499976c13c7a0f22e63, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.598839 906 consensus_queue.cc:260] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.599038 908 raft_consensus.cc:348] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.599121 908 raft_consensus.cc:374] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.599149 906 ts_tablet_manager.cc:1398] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240430 09:30:47.599150 908 raft_consensus.cc:729] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960dc67926f24ed78bf460935ffd13cd, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.599282 906 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63: Bootstrap starting.
I20240430 09:30:47.599262 908 consensus_queue.cc:260] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.599499 908 ts_tablet_manager.cc:1398] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240430 09:30:47.599539 907 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: Bootstrap starting.
I20240430 09:30:47.599645 908 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: Bootstrap starting.
I20240430 09:30:47.600153 906 tablet_bootstrap.cc:654] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.600548 908 tablet_bootstrap.cc:654] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.600626 907 tablet_bootstrap.cc:654] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.601210 906 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63: No bootstrap required, opened a new log
I20240430 09:30:47.601297 906 ts_tablet_manager.cc:1367] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240430 09:30:47.601790 907 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: No bootstrap required, opened a new log
I20240430 09:30:47.601744 906 raft_consensus.cc:348] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.601838 906 raft_consensus.cc:374] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.601876 906 raft_consensus.cc:729] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b0730cd16f4499976c13c7a0f22e63, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.601878 907 ts_tablet_manager.cc:1367] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240430 09:30:47.602010 906 consensus_queue.cc:260] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.602291 906 ts_tablet_manager.cc:1398] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240430 09:30:47.602466 907 raft_consensus.cc:348] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.602562 907 raft_consensus.cc:374] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.602594 907 raft_consensus.cc:729] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0d750046af4e748b72bfba40f38fbe, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.602622 908 tablet_bootstrap.cc:492] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: No bootstrap required, opened a new log
I20240430 09:30:47.602677 908 ts_tablet_manager.cc:1367] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240430 09:30:47.602713 907 consensus_queue.cc:260] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.602993 907 ts_tablet_manager.cc:1398] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240430 09:30:47.603081 908 raft_consensus.cc:348] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.603194 908 raft_consensus.cc:374] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.603230 908 raft_consensus.cc:729] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960dc67926f24ed78bf460935ffd13cd, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.603345 907 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: Bootstrap starting.
I20240430 09:30:47.603341 908 consensus_queue.cc:260] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.603546 908 ts_tablet_manager.cc:1398] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240430 09:30:47.604275 907 tablet_bootstrap.cc:654] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: Neither blocks nor log segments found. Creating new log.
I20240430 09:30:47.605937 907 tablet_bootstrap.cc:492] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: No bootstrap required, opened a new log
I20240430 09:30:47.606046 907 ts_tablet_manager.cc:1367] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240430 09:30:47.606503 907 raft_consensus.cc:348] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.606606 907 raft_consensus.cc:374] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:30:47.606650 907 raft_consensus.cc:729] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0d750046af4e748b72bfba40f38fbe, State: Initialized, Role: FOLLOWER
I20240430 09:30:47.606843 907 consensus_queue.cc:260] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.607137 907 ts_tablet_manager.cc:1398] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240430 09:30:47.671650 628 tablet.cc:2369] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:30:47.707654 912 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:30:47.708312 912 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.709605 912 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
I20240430 09:30:47.713672 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fef7488e904e04a08ab7025b4e4409" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd" is_pre_election: true
I20240430 09:30:47.714049 581 raft_consensus.cc:2454] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 0.
I20240430 09:30:47.714321 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fef7488e904e04a08ab7025b4e4409" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b0730cd16f4499976c13c7a0f22e63" is_pre_election: true
I20240430 09:30:47.714547 648 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd, ac0d750046af4e748b72bfba40f38fbe; no voters:
I20240430 09:30:47.714573 843 raft_consensus.cc:2454] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 0.
I20240430 09:30:47.714802 912 raft_consensus.cc:2781] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:30:47.714929 912 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:30:47.714993 912 raft_consensus.cc:3037] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.715853 912 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.716198 912 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 election: Requested vote from peers 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
I20240430 09:30:47.716508 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fef7488e904e04a08ab7025b4e4409" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd"
I20240430 09:30:47.716540 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28fef7488e904e04a08ab7025b4e4409" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b0730cd16f4499976c13c7a0f22e63"
I20240430 09:30:47.716631 581 raft_consensus.cc:3037] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.716643 843 raft_consensus.cc:3037] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.717711 581 raft_consensus.cc:2454] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 1.
I20240430 09:30:47.717711 843 raft_consensus.cc:2454] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 1.
I20240430 09:30:47.718037 647 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe; no voters:
I20240430 09:30:47.718243 912 raft_consensus.cc:2781] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:30:47.718436 912 raft_consensus.cc:686] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [term 1 LEADER]: Becoming Leader. State: Replica: ac0d750046af4e748b72bfba40f38fbe, State: Running, Role: LEADER
I20240430 09:30:47.718722 912 consensus_queue.cc:237] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.721589 439 catalog_manager.cc:5484] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to ac0d750046af4e748b72bfba40f38fbe (127.0.99.2). New cstate: current_term: 1 leader_uuid: "ac0d750046af4e748b72bfba40f38fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240430 09:30:47.736819 912 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:30:47.736948 912 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.737361 912 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385)
I20240430 09:30:47.737826 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b457e347ac7472291ba6d38c7cfb90c" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b0730cd16f4499976c13c7a0f22e63" is_pre_election: true
I20240430 09:30:47.737982 843 raft_consensus.cc:2454] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 0.
I20240430 09:30:47.738214 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b457e347ac7472291ba6d38c7cfb90c" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd" is_pre_election: true
I20240430 09:30:47.738281 647 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe; no voters:
I20240430 09:30:47.738363 581 raft_consensus.cc:2454] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 0.
I20240430 09:30:47.738435 912 raft_consensus.cc:2781] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:30:47.738495 912 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:30:47.738540 912 raft_consensus.cc:3037] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.739349 912 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.739663 912 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 election: Requested vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385)
I20240430 09:30:47.740002 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b457e347ac7472291ba6d38c7cfb90c" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd"
I20240430 09:30:47.740121 581 raft_consensus.cc:3037] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.740830 581 raft_consensus.cc:2454] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 1.
I20240430 09:30:47.741068 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b457e347ac7472291ba6d38c7cfb90c" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b0730cd16f4499976c13c7a0f22e63"
I20240430 09:30:47.741098 648 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd, ac0d750046af4e748b72bfba40f38fbe; no voters:
I20240430 09:30:47.741169 843 raft_consensus.cc:3037] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.741262 912 raft_consensus.cc:2781] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:30:47.741333 912 raft_consensus.cc:686] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [term 1 LEADER]: Becoming Leader. State: Replica: ac0d750046af4e748b72bfba40f38fbe, State: Running, Role: LEADER
I20240430 09:30:47.741480 912 consensus_queue.cc:237] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:47.742066 843 raft_consensus.cc:2454] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0d750046af4e748b72bfba40f38fbe in term 1.
I20240430 09:30:47.743355 439 catalog_manager.cc:5484] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to ac0d750046af4e748b72bfba40f38fbe (127.0.99.2). New cstate: current_term: 1 leader_uuid: "ac0d750046af4e748b72bfba40f38fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } health_report { overall_health: UNKNOWN } } }
W20240430 09:30:47.782704 890 tablet.cc:2369] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:30:47.834604 913 raft_consensus.cc:482] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:30:47.834753 913 raft_consensus.cc:504] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.835817 913 leader_election.cc:288] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
I20240430 09:30:47.839395 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "40b0730cd16f4499976c13c7a0f22e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd" is_pre_election: true
I20240430 09:30:47.839550 581 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b0730cd16f4499976c13c7a0f22e63 in term 0.
I20240430 09:30:47.839744 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "40b0730cd16f4499976c13c7a0f22e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0d750046af4e748b72bfba40f38fbe" is_pre_election: true
I20240430 09:30:47.839895 779 leader_election.cc:302] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b0730cd16f4499976c13c7a0f22e63, 960dc67926f24ed78bf460935ffd13cd; no voters:
I20240430 09:30:47.839956 712 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b0730cd16f4499976c13c7a0f22e63 in term 0.
I20240430 09:30:47.840193 913 raft_consensus.cc:2781] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:30:47.840282 913 raft_consensus.cc:482] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:30:47.840334 913 raft_consensus.cc:3037] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.841214 913 raft_consensus.cc:504] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.841552 913 leader_election.cc:288] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [CANDIDATE]: Term 1 election: Requested vote from peers 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
I20240430 09:30:47.841897 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "40b0730cd16f4499976c13c7a0f22e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd"
I20240430 09:30:47.841974 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "40b0730cd16f4499976c13c7a0f22e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0d750046af4e748b72bfba40f38fbe"
I20240430 09:30:47.842051 581 raft_consensus.cc:3037] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.842070 712 raft_consensus.cc:3037] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:30:47.843050 712 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b0730cd16f4499976c13c7a0f22e63 in term 1.
I20240430 09:30:47.843050 581 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b0730cd16f4499976c13c7a0f22e63 in term 1.
I20240430 09:30:47.843333 779 leader_election.cc:302] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe; no voters:
I20240430 09:30:47.843488 913 raft_consensus.cc:2781] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:30:47.843711 913 raft_consensus.cc:686] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [term 1 LEADER]: Becoming Leader. State: Replica: 40b0730cd16f4499976c13c7a0f22e63, State: Running, Role: LEADER
I20240430 09:30:47.844070 913 consensus_queue.cc:237] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:47.846999 439 catalog_manager.cc:5484] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "40b0730cd16f4499976c13c7a0f22e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240430 09:30:47.847810 759 tablet.cc:2369] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:30:47.856074 396 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:30:47.857692 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 960dc67926f24ed78bf460935ffd13cd to finish bootstrapping
I20240430 09:30:47.869017 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver ac0d750046af4e748b72bfba40f38fbe to finish bootstrapping
I20240430 09:30:47.873847 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 40b0730cd16f4499976c13c7a0f22e63 to finish bootstrapping
I20240430 09:30:47.878687 929 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.0.99.62:45235/tablet-servers
http://127.0.99.2:41881/maintenance-manager
http://127.0.99.2:41881/mem-trackers
http://127.0.99.3:33375/scans
http://127.0.99.1:45933/tablets
http://127.0.99.62:45235/startup
http://127.0.99.3:33375/metrics
http://127.0.99.62:45235/threadz?group=all
http://127.0.99.2:41881/transactions?tablet_id=28fef7488e904e04a08ab7025b4e4409
http://127.0.99.3:33375/transactions?tablet_id=28fef7488e904e04a08ab7025b4e4409
http://127.0.99.62:45235/mem-trackers
http://127.0.99.1:45933/threadz?group=all
http://127.0.99.2:41881/threadz?group=all
http://127.0.99.62:45235/ipki-ca-cert
http://127.0.99.62:45235/tables
http://127.0.99.2:41881/scans
http://127.0.99.62:45235/metrics
http://127.0.99.3:33375/startup
http://127.0.99.62:45235/masters
http://127.0.99.62:45235/dump-entities
http://127.0.99.2:41881/tablets
http://127.0.99.2:41881/stacks
http://127.0.99.1:45933/stacks
http://127.0.99.2:41881/metrics
http://127.0.99.3:33375/threadz
http://127.0.99.3:33375/mem-trackers
http://127.0.99.1:45933/startup
http://127.0.99.1:45933/threadz
http://127.0.99.62:45235/stacks
http://127.0.99.62:45235/ipki-ca-cert-der
http://127.0.99.1:45933/maintenance-manager
http://127.0.99.3:33375/tablets
http://127.0.99.3:33375/maintenance-manager
http://127.0.99.62:45235/threadz
http://127.0.99.1:45933/scans
http://127.0.99.2:41881/threadz
http://127.0.99.1:45933/metrics
http://127.0.99.62:45235/ipki-ca-cert-pem
http://127.0.99.2:41881/startup
http://127.0.99.3:33375/threadz?group=all
http://127.0.99.1:45933/transactions?tablet_id=28fef7488e904e04a08ab7025b4e4409
http://127.0.99.3:33375/stacks
http://127.0.99.1:45933/mem-trackers
I20240430 09:30:47.881726 396 linked_list-test-util.h:401] Running for: 30.000s
I20240430 09:30:47.881783 396 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 09:30:47.881799 396 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240430 09:30:47.887878 933 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:30:47.898361 712 raft_consensus.cc:1261] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Refusing update from remote peer 40b0730cd16f4499976c13c7a0f22e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.898465 581 raft_consensus.cc:1261] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Refusing update from remote peer 40b0730cd16f4499976c13c7a0f22e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.898898 913 consensus_queue.cc:1035] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.899055 926 consensus_queue.cc:1035] T 9f72c630f2044d14b15ef5d55ebec76f P 40b0730cd16f4499976c13c7a0f22e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.899962 581 raft_consensus.cc:1261] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Refusing update from remote peer ac0d750046af4e748b72bfba40f38fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.900370 843 raft_consensus.cc:1261] T 28fef7488e904e04a08ab7025b4e4409 P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Refusing update from remote peer ac0d750046af4e748b72bfba40f38fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.900382 842 raft_consensus.cc:1261] T 5b457e347ac7472291ba6d38c7cfb90c P 40b0730cd16f4499976c13c7a0f22e63 [term 1 FOLLOWER]: Refusing update from remote peer ac0d750046af4e748b72bfba40f38fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.900422 922 consensus_queue.cc:1035] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.902166 912 consensus_queue.cc:1035] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.902436 912 consensus_queue.cc:1035] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.908656 947 mvcc.cc:205] Tried to move back new op lower bound from 7022466858585206784 to 7022466858371821568. Current Snapshot: MvccSnapshot[applied={T|T < 7022466858585206784 or (T in {7022466858585206784})}]
I20240430 09:30:47.911361 943 mvcc.cc:205] Tried to move back new op lower bound from 7022466858592432128 to 7022466857950429184. Current Snapshot: MvccSnapshot[applied={T|T < 7022466858592432128}]
I20240430 09:30:47.916128 938 mvcc.cc:205] Tried to move back new op lower bound from 7022466858592432128 to 7022466857950429184. Current Snapshot: MvccSnapshot[applied={T|T < 7022466858592432128}]
I20240430 09:30:47.917207 581 raft_consensus.cc:1261] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Refusing update from remote peer ac0d750046af4e748b72bfba40f38fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:30:47.917757 922 consensus_queue.cc:1035] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:30:47.999084 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:30:49.852598 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:30:50.554512 933 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240430 09:30:50.591315 933 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f89db0dedcd google::LogMessage::Fail() at ??:0
@ 0x7f89db0e2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f89db0de7cc google::LogMessage::Flush() at ??:0
@ 0x7f89db0dff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f89dc4c05b1 kudu::operator-() at ??:0
@ 0x7f89de5a3fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f89de58d05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f89de58cf90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f89de5639bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f89df18f85d kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f89df18e936 _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7f89df197145 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7f89da059d09 std::function<>::operator()() at ??:0
@ 0x7f89da054107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f89da053a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f89da053123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f89da07bb89 handle_request at ??:0
@ 0x7f89da07e26b worker_thread at ??:0
@ 0x7f89dcd5d6db start_thread at ??:0
@ 0x7f89da29571f clone at ??:0
W20240430 09:30:50.806061 894 meta_cache.cc:302] tablet 9f72c630f2044d14b15ef5d55ebec76f: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: recv got EOF from 127.0.99.3:42655 (error 108)
W20240430 09:30:50.816812 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:50.820524 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
I20240430 09:30:50.829710 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:50.834923 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:50.845431 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:50.867120 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:50.879874 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:50.907604 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:50.924234 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:50.956514 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:50.973335 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.000106 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:30:51.004143 647 consensus_peers.cc:485] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 09:30:51.012995 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.015520 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.020251 647 consensus_peers.cc:485] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 09:30:51.030858 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.031677 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.041064 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.052235 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.064276 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.075984 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.096359 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.099504 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.110785 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.122164 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.142673 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.160831 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.170586 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.196493 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.198426 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.217115 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.249922 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.258185 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.281945 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.281945 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.331944 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.350409 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.355931 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.384747 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.414150 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.444684 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.458647 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.495849 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.504534 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.538828 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.575922 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.605329 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.617121 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.638964 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.700781 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.715171 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.743860 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.751384 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.829704 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.832616 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.870263 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.879782 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:51.914708 894 meta_cache.cc:302] tablet 9f72c630f2044d14b15ef5d55ebec76f: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:30:51.956980 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:51.973249 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:52.001052 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
W20240430 09:30:52.020890 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
I20240430 09:30:52.083398 997 raft_consensus.cc:482] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40b0730cd16f4499976c13c7a0f22e63)
I20240430 09:30:52.083542 997 raft_consensus.cc:504] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:52.083958 997 leader_election.cc:288] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960dc67926f24ed78bf460935ffd13cd (127.0.99.1:44385), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
I20240430 09:30:52.084264 579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "ac0d750046af4e748b72bfba40f38fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 187 } } ignore_live_leader: false dest_uuid: "960dc67926f24ed78bf460935ffd13cd" is_pre_election: true
W20240430 09:30:52.085212 647 leader_election.cc:334] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:52.085283 647 leader_election.cc:302] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac0d750046af4e748b72bfba40f38fbe; no voters: 40b0730cd16f4499976c13c7a0f22e63, 960dc67926f24ed78bf460935ffd13cd
I20240430 09:30:52.085436 997 raft_consensus.cc:2726] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:30:52.086712 1000 raft_consensus.cc:482] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40b0730cd16f4499976c13c7a0f22e63)
I20240430 09:30:52.086805 1000 raft_consensus.cc:504] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:52.087823 1000 leader_election.cc:288] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:30:52.088508 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:30:52.089267 516 leader_election.cc:334] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:52.092577 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "960dc67926f24ed78bf460935ffd13cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 187 } } ignore_live_leader: false dest_uuid: "ac0d750046af4e748b72bfba40f38fbe" is_pre_election: true
I20240430 09:30:52.092703 712 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 960dc67926f24ed78bf460935ffd13cd in term 1.
I20240430 09:30:52.093005 517 leader_election.cc:302] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd, ac0d750046af4e748b72bfba40f38fbe; no voters: 40b0730cd16f4499976c13c7a0f22e63
I20240430 09:30:52.093226 1000 raft_consensus.cc:2781] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 09:30:52.093281 1000 raft_consensus.cc:482] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40b0730cd16f4499976c13c7a0f22e63)
I20240430 09:30:52.093317 1000 raft_consensus.cc:3037] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Advancing to term 2
W20240430 09:30:52.094193 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:52.120829 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:52.125880 1000 raft_consensus.cc:504] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:52.126760 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72c630f2044d14b15ef5d55ebec76f" candidate_uuid: "960dc67926f24ed78bf460935ffd13cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 187 } } ignore_live_leader: false dest_uuid: "ac0d750046af4e748b72bfba40f38fbe"
I20240430 09:30:52.126897 712 raft_consensus.cc:3037] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 1 FOLLOWER]: Advancing to term 2
I20240430 09:30:52.126982 1000 leader_election.cc:288] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 election: Requested vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:30:52.127640 516 leader_election.cc:334] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:52.128010 712 raft_consensus.cc:2454] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 960dc67926f24ed78bf460935ffd13cd in term 2.
I20240430 09:30:52.128319 517 leader_election.cc:302] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd, ac0d750046af4e748b72bfba40f38fbe; no voters: 40b0730cd16f4499976c13c7a0f22e63
I20240430 09:30:52.128482 1000 raft_consensus.cc:2781] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 2 FOLLOWER]: Leader election won for term 2
I20240430 09:30:52.128705 1000 raft_consensus.cc:686] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [term 2 LEADER]: Becoming Leader. State: Replica: 960dc67926f24ed78bf460935ffd13cd, State: Running, Role: LEADER
I20240430 09:30:52.129057 1000 consensus_queue.cc:237] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 1.187, Last appended by leader: 187, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:52.131520 435 catalog_manager.cc:5484] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd reported cstate change: term changed from 1 to 2, leader changed from 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3) to 960dc67926f24ed78bf460935ffd13cd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "960dc67926f24ed78bf460935ffd13cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240430 09:30:52.140424 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
I20240430 09:30:52.171779 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:52.173142 671 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42156: Illegal state: replica ac0d750046af4e748b72bfba40f38fbe is not leader of this config: current role FOLLOWER
I20240430 09:30:52.198627 712 raft_consensus.cc:1261] T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe [term 2 FOLLOWER]: Refusing update from remote peer 960dc67926f24ed78bf460935ffd13cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 187. Preceding OpId from leader: term: 2 index: 189. (index mismatch)
I20240430 09:30:52.199209 1000 consensus_queue.cc:1035] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
W20240430 09:30:52.199807 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 09:30:52.238070 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:52.829382 1013 log.cc:924] Time spent T 9f72c630f2044d14b15ef5d55ebec76f P ac0d750046af4e748b72bfba40f38fbe: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20240430 09:30:52.829602 1023 log.cc:924] Time spent T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20240430 09:30:52.829841 1018 log.cc:924] Time spent T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20240430 09:30:52.830018 1020 log.cc:924] Time spent T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240430 09:30:52.830255 1006 log.cc:924] Time spent T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20240430 09:30:52.830256 1025 log.cc:924] Time spent T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240430 09:30:53.106657 759 maintenance_manager.cc:419] P ac0d750046af4e748b72bfba40f38fbe: Scheduling FlushMRSOp(5b457e347ac7472291ba6d38c7cfb90c): perf score=1.000000
I20240430 09:30:53.180438 628 maintenance_manager.cc:419] P 960dc67926f24ed78bf460935ffd13cd: Scheduling FlushMRSOp(5b457e347ac7472291ba6d38c7cfb90c): perf score=1.000000
W20240430 09:30:53.500347 647 consensus_peers.cc:485] T 5b457e347ac7472291ba6d38c7cfb90c P ac0d750046af4e748b72bfba40f38fbe -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:30:53.586486 649 maintenance_manager.cc:643] P ac0d750046af4e748b72bfba40f38fbe: FlushMRSOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.479s user 0.307s sys 0.004s Metrics: {"bytes_written":14648,"cfile_init":1,"compiler_manager_pool.queue_time_us":1463,"delete_count":0,"dirs.queue_time_us":12497,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":100631,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2214,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8491,"thread_start_us":237,"threads_started":2,"update_count":2269}
I20240430 09:30:53.587718 759 maintenance_manager.cc:419] P ac0d750046af4e748b72bfba40f38fbe: Scheduling MajorDeltaCompactionOp(5b457e347ac7472291ba6d38c7cfb90c): perf score=0.108769
W20240430 09:30:53.718761 647 consensus_peers.cc:485] T 28fef7488e904e04a08ab7025b4e4409 P ac0d750046af4e748b72bfba40f38fbe -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:30:53.753538 649 maintenance_manager.cc:643] P ac0d750046af4e748b72bfba40f38fbe: MajorDeltaCompactionOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.166s user 0.044s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15769,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":347,"lbm_reads_lt_1ms":25,"lbm_write_time_us":291,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":114,"threads_started":1,"update_count":2269}
I20240430 09:30:53.754067 759 maintenance_manager.cc:419] P ac0d750046af4e748b72bfba40f38fbe: Scheduling UndoDeltaBlockGCOp(5b457e347ac7472291ba6d38c7cfb90c): 132402 bytes on disk
I20240430 09:30:53.754631 649 maintenance_manager.cc:643] P ac0d750046af4e748b72bfba40f38fbe: UndoDeltaBlockGCOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240430 09:30:53.840662 518 maintenance_manager.cc:643] P 960dc67926f24ed78bf460935ffd13cd: FlushMRSOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.660s user 0.308s sys 0.008s Metrics: {"bytes_written":15704,"cfile_init":1,"compiler_manager_pool.queue_time_us":102622,"compiler_manager_pool.run_cpu_time_us":254150,"compiler_manager_pool.run_wall_time_us":594125,"delete_count":0,"dirs.queue_time_us":3753,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":36681,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1888,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8661,"thread_start_us":268,"threads_started":2,"update_count":2439}
I20240430 09:30:53.841780 628 maintenance_manager.cc:419] P 960dc67926f24ed78bf460935ffd13cd: Scheduling UndoDeltaBlockGCOp(5b457e347ac7472291ba6d38c7cfb90c): 134302 bytes on disk
I20240430 09:30:53.842411 518 maintenance_manager.cc:643] P 960dc67926f24ed78bf460935ffd13cd: UndoDeltaBlockGCOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240430 09:30:53.842866 628 maintenance_manager.cc:419] P 960dc67926f24ed78bf460935ffd13cd: Scheduling MajorDeltaCompactionOp(5b457e347ac7472291ba6d38c7cfb90c): perf score=0.114454
I20240430 09:30:53.912168 518 maintenance_manager.cc:643] P 960dc67926f24ed78bf460935ffd13cd: MajorDeltaCompactionOp(5b457e347ac7472291ba6d38c7cfb90c) complete. Timing: real 0.069s user 0.041s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":16838,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":34,"lbm_read_time_us":222,"lbm_reads_lt_1ms":21,"lbm_write_time_us":283,"lbm_writes_lt_1ms":9,"mutex_wait_us":23,"reinsert_count":0,"spinlock_wait_cycles":4864,"thread_start_us":71,"threads_started":1,"update_count":2439}
I20240430 09:30:54.289870 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240430 09:30:54.341233 932 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fe22530bdcd google::LogMessage::Fail() at ??:0
@ 0x7fe22530fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe22530b7cc google::LogMessage::Flush() at ??:0
@ 0x7fe22530cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe2266ed5b1 kudu::operator-() at ??:0
@ 0x7fe2287d0fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fe2287ba05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fe2287b9f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fe2287909bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fe2293bc85d kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fe2293bb936 _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fe2293c4145 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fe224286d09 std::function<>::operator()() at ??:0
@ 0x7fe224281107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fe224280a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fe224280123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fe2242a8b89 handle_request at ??:0
@ 0x7fe2242ab26b worker_thread at ??:0
@ 0x7fe226f8a6db start_thread at ??:0
@ 0x7fe2244c271f clone at ??:0
W20240430 09:30:54.567024 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
W20240430 09:30:54.576282 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.601881 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.639106 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.683427 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.732728 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.778198 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 09:30:54.790091 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.862269 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.938475 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:54.938848 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 09:30:55.027215 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.120811 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.224169 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.326043 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.338584 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.339152 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.343377 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.370478 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.380220 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.422329 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.422389 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:55.430262 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:55.483114 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.485723 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.522681 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.566665 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.588801 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:55.614456 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.636500 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.684515 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.717597 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.734009 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.832042 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:55.832695 1072 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:30:55.832827 1072 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:55.833354 1072 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:30:55.834302 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:55.834640 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:55.834718 516 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:30:55.834879 1072 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:30:55.836243 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:55.856485 1072 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:30:55.856613 1072 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:55.857036 1072 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:30:55.858156 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:55.858333 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:55.858393 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:30:55.858541 1072 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:30:55.899673 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.900756 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:55.989404 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.047200 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.128966 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.207046 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:56.298666 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:56.323791 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.336009 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.444067 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.471736 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.476644 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.569928 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.618938 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:56.699430 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.721225 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.831214 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:56.970335 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.001145 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.113225 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.243181 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 09:30:57.243930 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 09:30:57.259405 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:57.269493 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:57.301698 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:57.386953 1072 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:30:57.387099 1072 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:57.387504 1072 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:30:57.388598 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:57.388684 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:57.388743 516 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:30:57.388867 1072 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:30:57.404443 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.415288 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 09:30:57.499593 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.627075 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:57.703279 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.715739 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.844327 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:57.953693 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:57.975279 1072 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:30:57.975406 1072 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:30:57.975863 1072 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:30:57.977123 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:30:57.977205 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:30:57.977250 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:30:57.977401 1072 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:30:58.066861 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.201390 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:58.279209 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:58.294755 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.453722 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.552578 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.642688 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:30:58.699378 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.823426 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:58.953042 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.104329 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.217034 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:30:59.279937 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:30:59.485121 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.498713 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.607784 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 09:30:59.669642 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.697227 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:30:59.741519 1072 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:30:59.741658 1072 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:30:59.742142 1072 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:30:59.743220 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:30:59.743355 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:30:59.743466 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:30:59.743623 1072 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:30:59.799398 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:30:59.932968 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 09:30:59.953574 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:00.111640 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:00.249070 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:00.275141 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:00.435762 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:00.549829 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:00.718638 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:00.758196 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:00.882200 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:00.949990 1072 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:00.950198 1072 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:00.950810 1072 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:00.951891 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:00.952023 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:00.952138 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:00.952301 1072 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:01.079386 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:01.228744 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:01.267385 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:01.287578 1079 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:01.287698 1079 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:01.288137 1079 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:01.289103 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:01.289248 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:01.289304 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:01.289460 1079 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:01.406978 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:01.578346 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:01.748544 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:01.810065 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:01.937853 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:02.093616 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:02.183524 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 09:31:02.265977 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:02.406836 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240430 09:31:02.407446 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 09:31:03.238394 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:03.276121 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:03.476563 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:03.478710 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:03.491999 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:03.717934 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:03.879115 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:04.087235 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:04.264412 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:04.273746 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:04.336468 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:04.464216 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:04.673611 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:04.779160 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 09:31:04.855198 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:05.089920 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:05.101083 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 09:31:05.160653 1079 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:05.160799 1079 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:05.161284 1079 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:05.162624 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:05.162711 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:05.162819 517 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:05.162993 1079 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:05.251829 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:05.273509 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:05.367933 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:05.507457 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:05.662886 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:06.073882 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:06.085131 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:06.281055 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:06.355976 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:06.378719 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:06.525434 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:06.791175 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:06.972721 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:07.226132 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:07.237576 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 09:31:07.269160 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:07.285701 1079 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:07.285830 1079 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:07.286258 1079 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:07.287293 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:07.287381 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:07.287460 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:07.287591 1079 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:07.413275 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:07.432685 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:07.559633 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240430 09:31:07.560388 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 09:31:07.669600 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:07.894943 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:07.988610 1079 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:07.988812 1079 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:07.989423 1079 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:07.990657 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:07.990795 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:07.990859 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:07.991010 1079 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:08.123984 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:08.273099 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:08.374047 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:08.430658 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:08.587519 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:08.857431 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:09.066399 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:09.288805 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:09.307348 1079 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:09.307497 1079 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:09.307904 1079 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:09.308964 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:09.309273 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:09.309343 517 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:09.309492 1079 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:09.354406 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:09.441095 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:09.548233 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:09.773073 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 09:31:09.863174 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:10.042274 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:10.161851 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 09:31:10.269243 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:10.376699 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:10.544052 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:10.725714 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:10.880085 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:11.075639 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:11.252929 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:11.397497 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:11.573602 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:11.608134 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:11.921469 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:12.152611 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:12.265331 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:12.269286 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240430 09:31:12.458704 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:12.494319 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 09:31:12.640410 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:12.704200 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:12.734222 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240430 09:31:13.002704 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:13.266916 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:13.318435 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:13.555971 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:13.647917 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:13.839541 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:14.118674 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:14.261622 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:14.419911 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:14.692698 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:14.778475 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:14.792527 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 09:31:15.010496 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:15.182163 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240430 09:31:15.267112 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:15.283648 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:15.608954 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:15.809274 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:15.854908 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:16.215152 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:16.299571 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:16.449710 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:16.831007 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:16.857090 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:17.061146 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:17.133723 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 09:31:17.282418 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:17.457454 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:17.680819 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:17.739408 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240430 09:31:17.774660 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 09:31:17.882313 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:18.093305 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:18.288353 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:18.518365 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:18.725718 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:18.938665 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:18.948472 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:19.269018 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:19.332628 1085 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:19.332777 1085 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:19.333180 1085 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:19.334333 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:19.334441 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:19.334520 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:19.334656 1085 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:19.364760 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:19.599421 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:19.694175 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 09:31:20.010107 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 09:31:20.012645 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:20.040421 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:20.261783 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:20.266896 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:20.675966 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:20.936072 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:21.120471 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:21.282230 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:21.343463 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:21.618398 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:22.018855 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:22.088326 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 09:31:22.141431 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:22.269219 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:22.308857 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:22.434741 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 09:31:22.703096 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:22.878803 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240430 09:31:23.010833 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:23.164275 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:23.295671 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:23.397537 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:23.719753 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:24.102134 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:24.199517 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:24.279742 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:24.428193 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 09:31:24.437801 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:24.684317 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240430 09:31:24.727838 1085 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:24.727993 1085 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:24.728464 1085 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:24.729646 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:24.729799 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:24.729857 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:24.730038 1085 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:24.815829 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:25.164350 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:25.293228 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:25.305649 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:25.533464 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:25.894773 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:26.262928 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:26.302836 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:26.392560 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:26.641934 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:27.004674 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:27.037531 1085 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:27.037683 1085 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:27.038198 1085 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:27.039314 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:27.039443 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:27.039501 516 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:27.039652 1085 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:27.141422 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240430 09:31:27.285984 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240430 09:31:27.286233 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:27.400266 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:27.404816 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:27.711787 1085 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:27.711952 1085 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:27.712404 1085 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:27.713616 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:27.713708 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:27.713793 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:27.713971 1085 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:27.747931 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:28.026957 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 26 similar messages]
W20240430 09:31:28.163713 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:28.293496 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:28.413394 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:28.503283 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:28.932891 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:29.269168 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:29.280902 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:29.434832 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:29.523370 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240430 09:31:29.670565 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 09:31:29.712186 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:30.049330 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:30.280328 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:30.476301 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:30.498775 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:30.832886 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:31.273825 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:31.297327 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:31.534718 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:31.625633 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:31.910612 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 09:31:32.106639 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:32.281152 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:32.430861 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:32.604804 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:32.630326 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240430 09:31:32.920729 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:33.207968 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 09:31:33.241247 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:33.300949 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:33.685986 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:33.747401 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:34.058804 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:34.228294 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240430 09:31:34.289582 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:34.582700 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:34.786761 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:34.887013 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:35.265275 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240430 09:31:35.280862 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:35.426421 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:35.729612 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:35.904377 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:36.278188 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:36.282786 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:36.577091 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:36.672829 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240430 09:31:37.041445 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:37.140338 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:37.288723 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:37.431592 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:37.841105 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 09:31:38.007587 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:38.197453 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:38.289057 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:38.303361 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:38.453743 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 09:31:38.883534 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:38.965366 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 09:31:39.182793 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:39.275694 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:39.369659 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:39.758757 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:40.069849 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:40.283843 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:40.470052 1096 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:40.470188 1096 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:40.470642 1096 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:40.471791 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:40.471885 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:40.471975 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:40.472103 1096 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:40.555234 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:40.648825 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:40.748260 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240430 09:31:40.968586 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:41.263403 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:41.376863 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240430 09:31:41.547984 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:41.570765 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:41.872360 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:42.262120 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:42.453888 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:42.663637 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:42.790804 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:43.170194 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240430 09:31:43.290181 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:43.368676 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:43.467815 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 09:31:43.674196 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:43.721913 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:44.088979 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240430 09:31:44.275701 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:44.289819 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:44.658773 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:44.821568 894 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:45.223335 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:45.331072 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:45.606055 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:45.640914 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240430 09:31:45.848246 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:46.163861 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:46.283835 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:31:46.480605 1104 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:46.480751 1104 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:46.481226 1104 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:46.482450 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:46.482540 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:46.482604 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:46.482774 1104 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 09:31:46.559064 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:46.585146 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240430 09:31:46.877465 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:47.119508 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:47.300453 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:47.522023 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:47.994310 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:48.081073 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:48.086617 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240430 09:31:48.297755 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:48.493209 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:48.566727 516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 09:31:49.055415 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:49.086601 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240430 09:31:49.147229 895 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:49.279997 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:49.474037 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:50.035092 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:50.274758 895 meta_cache.cc:302] tablet 28fef7488e904e04a08ab7025b4e4409: replica ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:50.287853 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:50.459296 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:50.692853 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240430 09:31:51.022935 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:51.280864 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:51.354629 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:51.453572 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:51.799158 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240430 09:31:52.019897 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
I20240430 09:31:52.305270 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:52.355234 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:52.460340 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:53.024631 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:53.073747 516 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Couldn't send request to peer 40b0730cd16f4499976c13c7a0f22e63. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240430 09:31:53.205894 1108 raft_consensus.cc:482] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:53.206091 1108 raft_consensus.cc:504] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } }
I20240430 09:31:53.206540 1108 leader_election.cc:288] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655), ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585)
W20240430 09:31:53.207679 517 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
W20240430 09:31:53.207974 516 leader_election.cc:334] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
I20240430 09:31:53.208065 516 leader_election.cc:302] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:53.208248 1108 raft_consensus.cc:2726] T 5b457e347ac7472291ba6d38c7cfb90c P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:31:53.298764 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:53.359338 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:53.476315 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:53.658634 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240430 09:31:54.034139 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33170: Illegal state: replica 960dc67926f24ed78bf460935ffd13cd is not leader of this config: current role FOLLOWER
W20240430 09:31:54.199493 517 consensus_peers.cc:485] T 9f72c630f2044d14b15ef5d55ebec76f P 960dc67926f24ed78bf460935ffd13cd -> Peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Couldn't send request to peer ac0d750046af4e748b72bfba40f38fbe. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240430 09:31:54.277319 1108 raft_consensus.cc:482] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac0d750046af4e748b72bfba40f38fbe)
I20240430 09:31:54.277454 1108 raft_consensus.cc:504] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960dc67926f24ed78bf460935ffd13cd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44385 } } peers { permanent_uuid: "ac0d750046af4e748b72bfba40f38fbe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33585 } } peers { permanent_uuid: "40b0730cd16f4499976c13c7a0f22e63" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42655 } }
I20240430 09:31:54.277859 1108 leader_election.cc:288] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585), 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655)
W20240430 09:31:54.279072 516 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655): Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:54.279273 517 leader_election.cc:334] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac0d750046af4e748b72bfba40f38fbe (127.0.99.2:33585): Network error: Client connection negotiation failed: client connection to 127.0.99.2:33585: connect: Connection refused (error 111)
I20240430 09:31:54.279352 517 leader_election.cc:302] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 960dc67926f24ed78bf460935ffd13cd; no voters: 40b0730cd16f4499976c13c7a0f22e63, ac0d750046af4e748b72bfba40f38fbe
I20240430 09:31:54.279541 1108 raft_consensus.cc:2726] T 28fef7488e904e04a08ab7025b4e4409 P 960dc67926f24ed78bf460935ffd13cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:31:54.298532 935 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:31:54.374650 894 meta_cache.cc:302] tablet 5b457e347ac7472291ba6d38c7cfb90c: replica 40b0730cd16f4499976c13c7a0f22e63 (127.0.99.3:42655) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:42655: connect: Connection refused (error 111)
W20240430 09:31:54.506291 895 batcher.cc:441] Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.506772 396 client-test-util.cc:49] 32 failed ops. First 10 errors follow
I20240430 09:31:54.506882 396 client-test-util.cc:55] Op INSERT int64 rand_key=-5561261995256578048, int64 link_to=2999681980034318336, int64 insert_ts=1714469454310995 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507072 396 client-test-util.cc:55] Op INSERT int64 rand_key=-1760017775828271103, int64 link_to=4637939476754792449, int64 insert_ts=1714469454311113 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507125 396 client-test-util.cc:55] Op INSERT int64 rand_key=2041085706111614978, int64 link_to=6275774761010266114, int64 insert_ts=1714469454311152 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507171 396 client-test-util.cc:55] Op INSERT int64 rand_key=-6762084222629707770, int64 link_to=-2619101770713071610, int64 insert_ts=1714469454311276 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507216 396 client-test-util.cc:55] Op INSERT int64 rand_key=-2960980740689821689, int64 link_to=-980844273992597497, int64 insert_ts=1714469454311306 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507262 396 client-test-util.cc:55] Op INSERT int64 rand_key=-4721139254006382584, int64 link_to=3656954465273905160, int64 insert_ts=1714469454311335 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507308 396 client-test-util.cc:55] Op INSERT int64 rand_key=-919754297089785847, int64 link_to=5294930487017668617, int64 insert_ts=1714469454311363 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507354 396 client-test-util.cc:55] Op INSERT int64 rand_key=2881208447361810442, int64 link_to=6933047246249852938, int64 insert_ts=1714469454311391 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507398 396 client-test-util.cc:55] Op INSERT int64 rand_key=-5921820743891222514, int64 link_to=-1961829285473484786, int64 insert_ts=1714469454311503 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
I20240430 09:31:54.507445 396 client-test-util.cc:55] Op INSERT int64 rand_key=-2120576524462915569, int64 link_to=-323571788753010673, int64 insert_ts=1714469454311533 had status Timed out: Failed to write batch of 32 ops to tablet 5b457e347ac7472291ba6d38c7cfb90c after 340 attempt(s): Failed to write to server: (no server available): Write(tablet: 5b457e347ac7472291ba6d38c7cfb90c, num_ops: 32, num_attempts: 340) passed its deadline: Network error: recv got EOF from 127.0.99.2:33585 (error 108)
F20240430 09:31:54.507512 396 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714469514 (unix time) try "date -d @1714469514" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7f003fca7280) from PID 396; stack trace: ***
@ 0x7f004bfbe980 (unknown) at ??:0
@ 0x7f0049c42fb7 gsignal at ??:0
@ 0x7f0049c44921 abort at ??:0
@ 0x7f004ab4fdcd google::LogMessage::Fail() at ??:0
@ 0x7f004ab53b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f004ab4f7cc google::LogMessage::Flush() at ??:0
@ 0x7f004ab50f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f004d2817f2 kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x5621ba869236 kudu::client::FlushSessionOrDie() at ??:0
@ 0x5621ba86ccaa kudu::LinkedListTester::LoadLinkedList() at ??:0
@ 0x5621ba8624ca kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at ??:0
@ 0x7f004ba460ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f004ba3abdb testing::Test::Run() at ??:0
@ 0x7f004ba3ad9d testing::TestInfo::Run() at ??:0
@ 0x7f004ba3b377 testing::TestSuite::Run() at ??:0
@ 0x7f004ba3b77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f004ba4660d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f004ba3ae63 testing::UnitTest::Run() at ??:0
@ 0x7f004d2931de RUN_ALL_TESTS() at ??:0
@ 0x7f004d29253d main at ??:0
@ 0x7f0049c25bf7 __libc_start_main at ??:0
@ 0x5621ba86034a _start at ??:0