[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:14:04.540170 4859 test_util.cc:268] Using random seed: 925794522
I20240502 14:14:04.552605 4859 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 14:14:04.552708 4859 linked_list-test.cc:101] --------------
I20240502 14:14:04.552752 4859 linked_list-test.cc:102] 50 chains
I20240502 14:14:04.552798 4859 linked_list-test.cc:103] 3 tablets
I20240502 14:14:04.552848 4859 linked_list-test.cc:104] Mutations on
I20240502 14:14:04.552887 4859 linked_list-test.cc:105] --------------
I20240502 14:14:04.553104 4859 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:14:04.553169 4859 ts_itest-base.cc:113] --------------
I20240502 14:14:04.553207 4859 ts_itest-base.cc:114] 3 tablet servers
I20240502 14:14:04.553241 4859 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:14:04.553292 4859 ts_itest-base.cc:116] --------------
2024-05-02T14:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:14:04Z Disabled control of system clock
I20240502 14:14:04.580963 4859 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1cj1ur/build/asan/bin/kudu
/tmp/dist-test-task1cj1ur/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.190.254:34135
--webserver_interface=127.4.190.254
--webserver_port=0
--builtin_ntp_servers=127.4.190.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.190.254:34135
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 14:14:04.911172 4871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:04.911510 4871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:04.911617 4871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:04.919148 4871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:14:04.919268 4871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:04.919314 4871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:14:04.919404 4871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:14:04.928596 4871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.190.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.190.254:34135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.190.254:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.4.190.254
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:04.929941 4871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:04.931691 4871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:04.943423 4876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:04.943568 4877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:04.944113 4879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:04.944890 4871 server_base.cc:1034] running on GCE node
I20240502 14:14:04.945701 4871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:04.947196 4871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:04.948408 4871 hybrid_clock.cc:648] HybridClock initialized: now 1714659244948361 us; error 37 us; skew 500 ppm
I20240502 14:14:04.948765 4871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:04.951172 4871 webserver.cc:460] Webserver started at http://127.4.190.254:44757/ using document root <none> and password file <none>
I20240502 14:14:04.951933 4871 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:04.952059 4871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:04.952373 4871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:04.955091 4871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data/instance:
uuid: "fd815e4912be4706963258f7af063be1"
format_stamp: "Formatted at 2024-05-02 14:14:04 on dist-test-slave-g8l0"
I20240502 14:14:04.955864 4871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal/instance:
uuid: "fd815e4912be4706963258f7af063be1"
format_stamp: "Formatted at 2024-05-02 14:14:04 on dist-test-slave-g8l0"
I20240502 14:14:04.961067 4871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 14:14:04.964433 4885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:04.965730 4871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:14:04.965909 4871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data,/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal
uuid: "fd815e4912be4706963258f7af063be1"
format_stamp: "Formatted at 2024-05-02 14:14:04 on dist-test-slave-g8l0"
I20240502 14:14:04.966091 4871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:04.984829 4871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:04.985733 4871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:04.986035 4871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:05.004581 4871 rpc_server.cc:307] RPC server started. Bound to: 127.4.190.254:34135
I20240502 14:14:05.004650 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.190.254:34135 every 8 connection(s)
I20240502 14:14:05.006104 4871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/data/info.pb
I20240502 14:14:05.008872 4859 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1cj1ur/build/asan/bin/kudu as pid 4871
I20240502 14:14:05.009138 4859 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/master-0/wal/instance
I20240502 14:14:05.010051 4937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:05.020589 4937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1: Bootstrap starting.
I20240502 14:14:05.023694 4937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:05.024827 4937 log.cc:824] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:05.028076 4937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1: No bootstrap required, opened a new log
I20240502 14:14:05.036367 4937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } }
I20240502 14:14:05.036743 4937 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:05.036854 4937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd815e4912be4706963258f7af063be1, State: Initialized, Role: FOLLOWER
I20240502 14:14:05.037436 4937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } }
I20240502 14:14:05.037710 4937 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:05.037844 4937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:05.038022 4937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:05.040400 4937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } }
I20240502 14:14:05.040838 4937 leader_election.cc:302] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd815e4912be4706963258f7af063be1; no voters:
I20240502 14:14:05.041502 4937 leader_election.cc:288] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:05.041633 4942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:05.042768 4942 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [term 1 LEADER]: Becoming Leader. State: Replica: fd815e4912be4706963258f7af063be1, State: Running, Role: LEADER
I20240502 14:14:05.043272 4942 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } }
I20240502 14:14:05.043639 4937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:05.045863 4944 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd815e4912be4706963258f7af063be1. Latest consensus state: current_term: 1 leader_uuid: "fd815e4912be4706963258f7af063be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } } }
I20240502 14:14:05.046167 4944 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:05.046700 4943 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd815e4912be4706963258f7af063be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd815e4912be4706963258f7af063be1" member_type: VOTER last_known_addr { host: "127.4.190.254" port: 34135 } } }
I20240502 14:14:05.046969 4943 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:05.048629 4947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:05.053949 4947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:05.062721 4947 catalog_manager.cc:1347] Generated new cluster ID: df0756e6843a4ef785bdb3dcd4cedb03
I20240502 14:14:05.062839 4947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:05.086644 4947 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:05.088471 4947 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:05.103298 4947 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fd815e4912be4706963258f7af063be1: Generated new TSK 0
I20240502 14:14:05.104563 4947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:05.118937 4859 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1cj1ur/build/asan/bin/kudu
/tmp/dist-test-task1cj1ur/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.190.193:0
--local_ip_for_outbound_sockets=127.4.190.193
--webserver_interface=127.4.190.193
--webserver_port=0
--tserver_master_addrs=127.4.190.254:34135
--builtin_ntp_servers=127.4.190.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 14:14:05.454046 4961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:05.454398 4961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:05.454597 4961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:05.462226 4961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:05.462424 4961 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 14:14:05.462491 4961 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 14:14:05.462602 4961 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 14:14:05.462730 4961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.190.193
I20240502 14:14:05.472158 4961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.190.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.4.190.193
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.190.254:34135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.190.193
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:05.473567 4961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:05.475375 4961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:05.487066 4966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.487641 4969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.488111 4967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:05.488771 4961 server_base.cc:1034] running on GCE node
I20240502 14:14:05.489387 4961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:05.490350 4961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:05.491566 4961 hybrid_clock.cc:648] HybridClock initialized: now 1714659245491516 us; error 46 us; skew 500 ppm
I20240502 14:14:05.491894 4961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:05.494135 4961 webserver.cc:460] Webserver started at http://127.4.190.193:38989/ using document root <none> and password file <none>
I20240502 14:14:05.494894 4961 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:05.495014 4961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:05.495399 4961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:05.498087 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data/instance:
uuid: "7f06ca17ec1e4032baf8ca7598689037"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.498823 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal/instance:
uuid: "7f06ca17ec1e4032baf8ca7598689037"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.503873 4961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 14:14:05.507158 4975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.508493 4961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:05.508656 4961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal
uuid: "7f06ca17ec1e4032baf8ca7598689037"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.508842 4961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:05.531453 4961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:05.532348 4961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:05.532641 4961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:05.533609 4961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:05.535132 4961 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:05.535236 4961 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.535365 4961 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:05.535439 4961 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 14:14:05.567689 4961 rpc_server.cc:307] RPC server started. Bound to: 127.4.190.193:34177
I20240502 14:14:05.567729 5091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.190.193:34177 every 8 connection(s)
I20240502 14:14:05.569047 4961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/data/info.pb
I20240502 14:14:05.578191 4859 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1cj1ur/build/asan/bin/kudu as pid 4961
I20240502 14:14:05.578446 4859 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-0/wal/instance
I20240502 14:14:05.582288 4859 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1cj1ur/build/asan/bin/kudu
/tmp/dist-test-task1cj1ur/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.190.194:0
--local_ip_for_outbound_sockets=127.4.190.194
--webserver_interface=127.4.190.194
--webserver_port=0
--tserver_master_addrs=127.4.190.254:34135
--builtin_ntp_servers=127.4.190.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 14:14:05.582584 5092 heartbeater.cc:347] Connected to a master server at 127.4.190.254:34135
I20240502 14:14:05.582875 5092 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:05.583755 5092 heartbeater.cc:511] Master 127.4.190.254:34135 requested a full tablet report, sending...
I20240502 14:14:05.585855 4902 ts_manager.cc:194] Registered new tserver with Master: 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
I20240502 14:14:05.587457 4902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.190.193:37487
W20240502 14:14:05.912076 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:05.912380 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:05.912545 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:05.920197 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:05.920351 5096 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 14:14:05.920428 5096 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 14:14:05.920501 5096 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 14:14:05.920631 5096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.190.194
I20240502 14:14:05.929792 5096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.190.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.4.190.194
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.190.254:34135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.190.194
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:05.931114 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:05.932905 5096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:05.945402 5102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.945506 5101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.945962 5104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:05.946725 5096 server_base.cc:1034] running on GCE node
I20240502 14:14:05.947374 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:05.948463 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:05.949721 5096 hybrid_clock.cc:648] HybridClock initialized: now 1714659245949661 us; error 51 us; skew 500 ppm
I20240502 14:14:05.950062 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:05.952497 5096 webserver.cc:460] Webserver started at http://127.4.190.194:41327/ using document root <none> and password file <none>
I20240502 14:14:05.953272 5096 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:05.953385 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:05.953724 5096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:05.956599 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data/instance:
uuid: "83db009b0a0a4b189a7bc71211a72921"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.957404 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal/instance:
uuid: "83db009b0a0a4b189a7bc71211a72921"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.962805 5096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 14:14:05.966456 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.967936 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:14:05.968098 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal
uuid: "83db009b0a0a4b189a7bc71211a72921"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-g8l0"
I20240502 14:14:05.968304 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:06.000658 5096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:06.001574 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:06.001883 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:06.002914 5096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:06.004544 5096 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:06.004645 5096 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:06.004765 5096 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:06.004843 5096 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:06.038050 5096 rpc_server.cc:307] RPC server started. Bound to: 127.4.190.194:44571
I20240502 14:14:06.038120 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.190.194:44571 every 8 connection(s)
I20240502 14:14:06.039393 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/data/info.pb
I20240502 14:14:06.040362 4859 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1cj1ur/build/asan/bin/kudu as pid 5096
I20240502 14:14:06.040562 4859 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-1/wal/instance
I20240502 14:14:06.044476 4859 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1cj1ur/build/asan/bin/kudu
/tmp/dist-test-task1cj1ur/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.190.195:0
--local_ip_for_outbound_sockets=127.4.190.195
--webserver_interface=127.4.190.195
--webserver_port=0
--tserver_master_addrs=127.4.190.254:34135
--builtin_ntp_servers=127.4.190.212:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 14:14:06.053349 5224 heartbeater.cc:347] Connected to a master server at 127.4.190.254:34135
I20240502 14:14:06.053710 5224 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:06.054803 5224 heartbeater.cc:511] Master 127.4.190.254:34135 requested a full tablet report, sending...
I20240502 14:14:06.056509 4902 ts_manager.cc:194] Registered new tserver with Master: 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
I20240502 14:14:06.057299 4902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.190.194:49425
W20240502 14:14:06.378926 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:06.379228 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:06.379376 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:06.386886 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:06.387042 5227 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 14:14:06.387091 5227 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 14:14:06.387151 5227 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 14:14:06.387234 5227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.190.195
I20240502 14:14:06.396499 5227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.190.212:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.4.190.195
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.4.190.254:34135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.190.195
--log_dir=/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:06.397857 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:06.399667 5227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:06.411530 5232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:06.412360 5227 server_base.cc:1034] running on GCE node
W20240502 14:14:06.412789 5233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:06.412878 5235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:06.413590 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:06.414577 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:06.415791 5227 hybrid_clock.cc:648] HybridClock initialized: now 1714659246415740 us; error 46 us; skew 500 ppm
I20240502 14:14:06.416108 5227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:06.418359 5227 webserver.cc:460] Webserver started at http://127.4.190.195:46603/ using document root <none> and password file <none>
I20240502 14:14:06.419114 5227 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:06.419243 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:06.419595 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:06.422266 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data/instance:
uuid: "32ace0a8418742349cdbbaaca6fbd128"
format_stamp: "Formatted at 2024-05-02 14:14:06 on dist-test-slave-g8l0"
I20240502 14:14:06.422986 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal/instance:
uuid: "32ace0a8418742349cdbbaaca6fbd128"
format_stamp: "Formatted at 2024-05-02 14:14:06 on dist-test-slave-g8l0"
I20240502 14:14:06.428180 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 14:14:06.431555 5241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:06.432873 5227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:14:06.433033 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal
uuid: "32ace0a8418742349cdbbaaca6fbd128"
format_stamp: "Formatted at 2024-05-02 14:14:06 on dist-test-slave-g8l0"
I20240502 14:14:06.433235 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:06.468008 5227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:06.468899 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:06.469202 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:06.470227 5227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:06.471889 5227 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:06.471992 5227 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:06.472103 5227 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:06.472180 5227 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:06.503703 5227 rpc_server.cc:307] RPC server started. Bound to: 127.4.190.195:43459
I20240502 14:14:06.503746 5354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.190.195:43459 every 8 connection(s)
I20240502 14:14:06.505033 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/data/info.pb
I20240502 14:14:06.514130 4859 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1cj1ur/build/asan/bin/kudu as pid 5227
I20240502 14:14:06.514369 4859 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1cj1ur/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714659244463064-4859-0/linked-list-cluster/ts-2/wal/instance
I20240502 14:14:06.518220 5355 heartbeater.cc:347] Connected to a master server at 127.4.190.254:34135
I20240502 14:14:06.518529 5355 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:06.519488 5355 heartbeater.cc:511] Master 127.4.190.254:34135 requested a full tablet report, sending...
I20240502 14:14:06.520787 4902 ts_manager.cc:194] Registered new tserver with Master: 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:06.521566 4902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.190.195:34685
I20240502 14:14:06.531072 4859 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:14:06.552407 4902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48026:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 14:14:06.558827 4902 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:14:06.590490 5092 heartbeater.cc:503] Master 127.4.190.254:34135 was elected leader, sending a full tablet report...
I20240502 14:14:06.601164 5289 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccdc3ce29154921b753cc603d8ded3e (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 14:14:06.602490 5288 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 14:14:06.602063 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc78b71a424b069d2c823e9be1bdd7 (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 14:14:06.602079 5157 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 14:14:06.603289 5289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccdc3ce29154921b753cc603d8ded3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.603895 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc78b71a424b069d2c823e9be1bdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.603842 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 14:14:06.603905 5158 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccdc3ce29154921b753cc603d8ded3e (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 14:14:06.604192 5024 tablet_service.cc:1467] Processing CreateTablet for tablet 4ccdc3ce29154921b753cc603d8ded3e (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 14:14:06.604440 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.604795 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccdc3ce29154921b753cc603d8ded3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.605248 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc78b71a424b069d2c823e9be1bdd7 (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 14:14:06.603924 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.605650 5287 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc78b71a424b069d2c823e9be1bdd7 (DEFAULT_TABLE table=TestTable [id=827db4823361470791bbfa89d08cc084]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 14:14:06.606318 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ccdc3ce29154921b753cc603d8ded3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.607056 5287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc78b71a424b069d2c823e9be1bdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.607373 5156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc78b71a424b069d2c823e9be1bdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.610955 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.626945 5374 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Bootstrap starting.
I20240502 14:14:06.628863 5373 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Bootstrap starting.
I20240502 14:14:06.631522 5374 tablet_bootstrap.cc:654] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.631798 5373 tablet_bootstrap.cc:654] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.632498 5375 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Bootstrap starting.
I20240502 14:14:06.633008 5374 log.cc:824] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:06.633123 5373 log.cc:824] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:06.636688 5375 tablet_bootstrap.cc:654] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.638037 5374 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: No bootstrap required, opened a new log
I20240502 14:14:06.638038 5375 log.cc:824] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:06.638249 5373 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: No bootstrap required, opened a new log
I20240502 14:14:06.638319 5374 ts_tablet_manager.cc:1367] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 14:14:06.638531 5373 ts_tablet_manager.cc:1367] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 14:14:06.641305 5375 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: No bootstrap required, opened a new log
I20240502 14:14:06.641597 5375 ts_tablet_manager.cc:1367] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 14:14:06.646215 5374 raft_consensus.cc:348] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.646613 5374 raft_consensus.cc:374] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.646723 5374 raft_consensus.cc:729] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ace0a8418742349cdbbaaca6fbd128, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.647462 5374 consensus_queue.cc:260] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.647536 5373 raft_consensus.cc:348] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.648042 5373 raft_consensus.cc:374] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.648157 5373 raft_consensus.cc:729] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83db009b0a0a4b189a7bc71211a72921, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.648859 5355 heartbeater.cc:503] Master 127.4.190.254:34135 was elected leader, sending a full tablet report...
I20240502 14:14:06.649224 5373 consensus_queue.cc:260] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.649523 5374 ts_tablet_manager.cc:1398] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:14:06.650197 5374 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: Bootstrap starting.
I20240502 14:14:06.652017 5374 tablet_bootstrap.cc:654] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.652120 5375 raft_consensus.cc:348] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.652711 5375 raft_consensus.cc:374] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.652855 5375 raft_consensus.cc:729] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.653860 5375 consensus_queue.cc:260] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.658900 5224 heartbeater.cc:503] Master 127.4.190.254:34135 was elected leader, sending a full tablet report...
I20240502 14:14:06.659922 5373 ts_tablet_manager.cc:1398] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20240502 14:14:06.660415 5373 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921: Bootstrap starting.
I20240502 14:14:06.661576 5375 ts_tablet_manager.cc:1398] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20240502 14:14:06.662274 5373 tablet_bootstrap.cc:654] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.663461 5375 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: Bootstrap starting.
I20240502 14:14:06.665583 5374 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: No bootstrap required, opened a new log
I20240502 14:14:06.665764 5374 ts_tablet_manager.cc:1367] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240502 14:14:06.666754 5374 raft_consensus.cc:348] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.667018 5374 raft_consensus.cc:374] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.667121 5374 raft_consensus.cc:729] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ace0a8418742349cdbbaaca6fbd128, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.667419 5374 consensus_queue.cc:260] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.667670 5375 tablet_bootstrap.cc:654] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.668077 5374 ts_tablet_manager.cc:1398] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:06.668334 5374 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Bootstrap starting.
I20240502 14:14:06.670967 5374 tablet_bootstrap.cc:654] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.673895 5375 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: No bootstrap required, opened a new log
I20240502 14:14:06.674067 5375 ts_tablet_manager.cc:1367] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 14:14:06.674485 5374 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: No bootstrap required, opened a new log
I20240502 14:14:06.674489 5373 tablet_bootstrap.cc:492] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921: No bootstrap required, opened a new log
I20240502 14:14:06.674634 5373 ts_tablet_manager.cc:1367] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 14:14:06.674644 5374 ts_tablet_manager.cc:1367] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:14:06.675144 5375 raft_consensus.cc:348] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.675206 5374 raft_consensus.cc:348] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.675407 5375 raft_consensus.cc:374] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.675423 5374 raft_consensus.cc:374] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.675498 5374 raft_consensus.cc:729] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ace0a8418742349cdbbaaca6fbd128, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.675518 5375 raft_consensus.cc:729] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.675630 5373 raft_consensus.cc:348] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.675745 5374 consensus_queue.cc:260] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.675858 5373 raft_consensus.cc:374] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.675947 5373 raft_consensus.cc:729] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83db009b0a0a4b189a7bc71211a72921, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.675832 5375 consensus_queue.cc:260] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.676198 5373 consensus_queue.cc:260] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.676420 5375 ts_tablet_manager.cc:1398] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:06.676672 5375 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Bootstrap starting.
I20240502 14:14:06.676707 5373 ts_tablet_manager.cc:1398] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:06.676960 5373 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921: Bootstrap starting.
I20240502 14:14:06.678517 5373 tablet_bootstrap.cc:654] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.678527 5375 tablet_bootstrap.cc:654] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:06.678894 5382 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:06.679147 5382 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.681499 5382 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:06.688526 5374 ts_tablet_manager.cc:1398] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:14:06.701373 5375 tablet_bootstrap.cc:492] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: No bootstrap required, opened a new log
I20240502 14:14:06.701579 5375 ts_tablet_manager.cc:1367] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20240502 14:14:06.701861 5373 tablet_bootstrap.cc:492] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921: No bootstrap required, opened a new log
I20240502 14:14:06.701658 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba0a6a6ce8945e4bfba8c6c31ef7dab" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128" is_pre_election: true
I20240502 14:14:06.702042 5373 ts_tablet_manager.cc:1367] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.004s
I20240502 14:14:06.702265 5309 raft_consensus.cc:2454] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 0.
I20240502 14:14:06.702399 5375 raft_consensus.cc:348] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.702672 5375 raft_consensus.cc:374] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.702773 5375 raft_consensus.cc:729] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.702699 5373 raft_consensus.cc:348] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.702965 5373 raft_consensus.cc:374] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:06.703049 5373 raft_consensus.cc:729] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83db009b0a0a4b189a7bc71211a72921, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.703028 5375 consensus_queue.cc:260] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.703166 4977 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 7f06ca17ec1e4032baf8ca7598689037; no voters:
I20240502 14:14:06.703320 5373 consensus_queue.cc:260] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.703774 5375 ts_tablet_manager.cc:1398] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:06.704533 5382 raft_consensus.cc:2781] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:06.704495 5178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba0a6a6ce8945e4bfba8c6c31ef7dab" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83db009b0a0a4b189a7bc71211a72921" is_pre_election: true
I20240502 14:14:06.704689 5382 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:06.704784 5382 raft_consensus.cc:3037] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.704993 5178 raft_consensus.cc:2454] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 0.
I20240502 14:14:06.705034 5373 ts_tablet_manager.cc:1398] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:06.707423 5382 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.707973 5382 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 1 election: Requested vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:06.708729 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba0a6a6ce8945e4bfba8c6c31ef7dab" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128"
I20240502 14:14:06.708808 5178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba0a6a6ce8945e4bfba8c6c31ef7dab" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83db009b0a0a4b189a7bc71211a72921"
I20240502 14:14:06.708935 5309 raft_consensus.cc:3037] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.709003 5178 raft_consensus.cc:3037] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.711297 5178 raft_consensus.cc:2454] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 1.
I20240502 14:14:06.711298 5309 raft_consensus.cc:2454] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 1.
I20240502 14:14:06.711733 4979 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921; no voters:
I20240502 14:14:06.712005 5382 raft_consensus.cc:2781] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:06.712486 5382 raft_consensus.cc:686] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [term 1 LEADER]: Becoming Leader. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Running, Role: LEADER
I20240502 14:14:06.712960 5382 consensus_queue.cc:237] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.717963 4900 catalog_manager.cc:5484] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193). New cstate: current_term: 1 leader_uuid: "7f06ca17ec1e4032baf8ca7598689037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:06.756758 5356 tablet.cc:2369] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:06.791673 5225 tablet.cc:2369] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:06.797430 5380 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:06.797667 5380 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.799379 5380 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
I20240502 14:14:06.807812 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128" is_pre_election: true
I20240502 14:14:06.808068 5309 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 0.
I20240502 14:14:06.808125 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037" is_pre_election: true
I20240502 14:14:06.808535 5044 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 0.
I20240502 14:14:06.808528 5112 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 83db009b0a0a4b189a7bc71211a72921; no voters:
I20240502 14:14:06.808974 5380 raft_consensus.cc:2781] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:06.809109 5380 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:06.809213 5380 raft_consensus.cc:3037] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.811256 5380 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.811870 5380 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 election: Requested vote from peers 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
I20240502 14:14:06.812337 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037"
I20240502 14:14:06.812340 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128"
I20240502 14:14:06.812513 5044 raft_consensus.cc:3037] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.812525 5309 raft_consensus.cc:3037] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.814549 5309 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 1.
I20240502 14:14:06.814553 5044 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 1.
I20240502 14:14:06.814970 5113 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921; no voters:
I20240502 14:14:06.815276 5380 raft_consensus.cc:2781] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:06.815805 5380 raft_consensus.cc:686] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [term 1 LEADER]: Becoming Leader. State: Replica: 83db009b0a0a4b189a7bc71211a72921, State: Running, Role: LEADER
I20240502 14:14:06.816346 5380 consensus_queue.cc:237] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:06.820809 4900 catalog_manager.cc:5484] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194). New cstate: current_term: 1 leader_uuid: "83db009b0a0a4b189a7bc71211a72921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:06.821431 5093 tablet.cc:2369] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:06.887631 5380 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:06.887887 5380 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.888548 5380 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:06.889133 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037" is_pre_election: true
I20240502 14:14:06.889146 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128" is_pre_election: true
I20240502 14:14:06.889348 5309 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 0.
I20240502 14:14:06.889382 5044 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 0.
I20240502 14:14:06.889803 5113 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921; no voters:
I20240502 14:14:06.890075 5380 raft_consensus.cc:2781] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:06.890164 5380 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:06.890241 5380 raft_consensus.cc:3037] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.892136 5380 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.892710 5380 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 election: Requested vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:06.893153 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037"
I20240502 14:14:06.893265 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "83db009b0a0a4b189a7bc71211a72921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128"
I20240502 14:14:06.893393 5044 raft_consensus.cc:3037] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.893493 5309 raft_consensus.cc:3037] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:06.896317 5044 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 1.
I20240502 14:14:06.896338 5309 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83db009b0a0a4b189a7bc71211a72921 in term 1.
I20240502 14:14:06.896759 5112 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 83db009b0a0a4b189a7bc71211a72921; no voters:
I20240502 14:14:06.897066 5380 raft_consensus.cc:2781] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:06.897218 5380 raft_consensus.cc:686] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [term 1 LEADER]: Becoming Leader. State: Replica: 83db009b0a0a4b189a7bc71211a72921, State: Running, Role: LEADER
I20240502 14:14:06.897466 5380 consensus_queue.cc:237] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:06.900298 4900 catalog_manager.cc:5484] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194). New cstate: current_term: 1 leader_uuid: "83db009b0a0a4b189a7bc71211a72921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:06.975032 4859 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:14:06.978061 4859 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 7f06ca17ec1e4032baf8ca7598689037 to finish bootstrapping
I20240502 14:14:06.988276 4859 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 83db009b0a0a4b189a7bc71211a72921 to finish bootstrapping
I20240502 14:14:06.996513 4859 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 32ace0a8418742349cdbbaaca6fbd128 to finish bootstrapping
I20240502 14:14:07.005514 5398 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.4.190.195:46603/threadz?group=all
http://127.4.190.193:38989/metrics
http://127.4.190.194:41327/threadz?group=all
http://127.4.190.254:44757/metrics
http://127.4.190.254:44757/tables
http://127.4.190.254:44757/startup
http://127.4.190.193:38989/tablets
http://127.4.190.193:38989/startup
http://127.4.190.195:46603/maintenance-manager
http://127.4.190.193:38989/threadz?group=all
http://127.4.190.193:38989/threadz
http://127.4.190.195:46603/metrics
http://127.4.190.195:46603/startup
http://127.4.190.195:46603/scans
http://127.4.190.194:41327/scans
http://127.4.190.194:41327/mem-trackers
http://127.4.190.254:44757/dump-entities
http://127.4.190.195:46603/mem-trackers
http://127.4.190.193:38989/stacks
http://127.4.190.254:44757/ipki-ca-cert-der
http://127.4.190.195:46603/tablets
http://127.4.190.193:38989/maintenance-manager
http://127.4.190.254:44757/tablet-servers
http://127.4.190.194:41327/stacks
http://127.4.190.195:46603/transactions?tablet_id=3fcc78b71a424b069d2c823e9be1bdd7
http://127.4.190.194:41327/threadz
http://127.4.190.254:44757/ipki-ca-cert
http://127.4.190.194:41327/tablets
http://127.4.190.194:41327/startup
http://127.4.190.194:41327/transactions?tablet_id=3fcc78b71a424b069d2c823e9be1bdd7
http://127.4.190.195:46603/threadz
http://127.4.190.195:46603/stacks
http://127.4.190.193:38989/mem-trackers
http://127.4.190.254:44757/masters
http://127.4.190.194:41327/maintenance-manager
http://127.4.190.254:44757/ipki-ca-cert-pem
http://127.4.190.193:38989/scans
http://127.4.190.254:44757/stacks
http://127.4.190.254:44757/threadz?group=all
http://127.4.190.194:41327/metrics
http://127.4.190.254:44757/threadz
http://127.4.190.193:38989/transactions?tablet_id=3fcc78b71a424b069d2c823e9be1bdd7
http://127.4.190.254:44757/mem-trackers
I20240502 14:14:07.011855 4859 linked_list-test-util.h:401] Running for: 30.000s
I20240502 14:14:07.012091 4859 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 14:14:07.012267 4859 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 14:14:07.044911 5309 raft_consensus.cc:1261] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Refusing update from remote peer 83db009b0a0a4b189a7bc71211a72921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.045080 5044 raft_consensus.cc:1261] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Refusing update from remote peer 83db009b0a0a4b189a7bc71211a72921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.045728 5308 raft_consensus.cc:1261] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Refusing update from remote peer 83db009b0a0a4b189a7bc71211a72921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.045900 5380 consensus_queue.cc:1035] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.046375 5394 consensus_queue.cc:1035] T 4ccdc3ce29154921b753cc603d8ded3e P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.046654 5044 raft_consensus.cc:1261] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Refusing update from remote peer 83db009b0a0a4b189a7bc71211a72921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.047003 5394 consensus_queue.cc:1035] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.047441 5380 consensus_queue.cc:1035] T 3fcc78b71a424b069d2c823e9be1bdd7 P 83db009b0a0a4b189a7bc71211a72921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.056128 5307 raft_consensus.cc:1261] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Refusing update from remote peer 7f06ca17ec1e4032baf8ca7598689037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.057574 5382 consensus_queue.cc:1035] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.058485 5178 raft_consensus.cc:1261] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 83db009b0a0a4b189a7bc71211a72921 [term 1 FOLLOWER]: Refusing update from remote peer 7f06ca17ec1e4032baf8ca7598689037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:07.059207 5382 consensus_queue.cc:1035] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:07.068567 5405 mvcc.cc:205] Tried to move back new op lower bound from 7023244275889405952 to 7023244275294146560. Current Snapshot: MvccSnapshot[applied={T|T < 7023244275889405952}]
I20240502 14:14:07.082063 5406 mvcc.cc:205] Tried to move back new op lower bound from 7023244275889405952 to 7023244275294146560. Current Snapshot: MvccSnapshot[applied={T|T < 7023244275889405952}]
I20240502 14:14:07.096540 5411 mvcc.cc:205] Tried to move back new op lower bound from 7023244275889405952 to 7023244275294146560. Current Snapshot: MvccSnapshot[applied={T|T < 7023244275889405952}]
I20240502 14:14:07.361940 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:07.363937 5430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:10.178625 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:11.184430 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:11.186758 5430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:12.520103 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:12.563517 5225 maintenance_manager.cc:419] P 83db009b0a0a4b189a7bc71211a72921: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:12.592166 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
W20240502 14:14:14.556778 5246 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.241s user 0.000s sys 0.000s
I20240502 14:14:14.572443 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.051s user 0.597s sys 0.112s Metrics: {"bytes_written":145992,"cfile_init":1,"compiler_manager_pool.queue_time_us":718470,"compiler_manager_pool.run_cpu_time_us":668424,"compiler_manager_pool.run_wall_time_us":1648084,"dirs.queue_time_us":85950,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":2613,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2022,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7927,"thread_start_us":110487,"threads_started":2}
I20240502 14:14:14.575440 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 139933 bytes on disk
I20240502 14:14:14.576676 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 14:14:14.822417 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.229s user 0.543s sys 0.095s Metrics: {"bytes_written":1630,"cfile_init":1,"compiler_manager_pool.queue_time_us":53015,"compiler_manager_pool.run_cpu_time_us":656899,"compiler_manager_pool.run_wall_time_us":1797295,"delete_count":0,"dirs.queue_time_us":5965,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":52006,"drs_written":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4,"lbm_write_time_us":34466,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7962,"thread_start_us":71594,"threads_started":2,"update_count":229}
I20240502 14:14:14.827260 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 140358 bytes on disk
I20240502 14:14:14.828310 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 14:14:14.829126 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=0.012935
I20240502 14:14:14.931741 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.102s user 0.069s sys 0.016s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2242,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":3775,"dirs.run_cpu_time_us":375,"dirs.run_wall_time_us":4536,"lbm_read_time_us":411,"lbm_reads_lt_1ms":20,"lbm_write_time_us":559,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":21196,"threads_started":1,"update_count":229}
I20240502 14:14:15.173923 5115 maintenance_manager.cc:643] P 83db009b0a0a4b189a7bc71211a72921: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.590s user 0.592s sys 0.119s Metrics: {"bytes_written":1630,"cfile_init":1,"compiler_manager_pool.queue_time_us":13771,"delete_count":0,"dirs.queue_time_us":9841,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":7668,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":34191,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7962,"thread_start_us":28908,"threads_started":2,"update_count":229}
I20240502 14:14:15.176007 5225 maintenance_manager.cc:419] P 83db009b0a0a4b189a7bc71211a72921: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 140358 bytes on disk
I20240502 14:14:15.177045 5115 maintenance_manager.cc:643] P 83db009b0a0a4b189a7bc71211a72921: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 14:14:15.180872 5225 maintenance_manager.cc:419] P 83db009b0a0a4b189a7bc71211a72921: Scheduling MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=0.012935
I20240502 14:14:15.297601 5115 maintenance_manager.cc:643] P 83db009b0a0a4b189a7bc71211a72921: MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.116s user 0.065s sys 0.024s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2242,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":499,"lbm_read_time_us":553,"lbm_reads_lt_1ms":20,"lbm_write_time_us":394,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":290,"threads_started":1,"update_count":229}
I20240502 14:14:15.653062 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:16.694346 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:16.698148 5430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:17.019809 4859 linked_list-test-util.h:417] Sample at HT timestamp: 7023244316712456192 Inserted count: 19400
F20240502 14:14:18.558797 5430 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fb99cb1edcd google::LogMessage::Fail() at ??:0
@ 0x7fb99cb22b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb99cb1e7cc google::LogMessage::Flush() at ??:0
@ 0x7fb99cb1ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb99f747c80 kudu::operator-() at ??:0
@ 0x7fb9ac3c814a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fb9ac3925de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fb9ac392336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fb9ac33f2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fb9aee0ce47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fb9aee1b9fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fb9aee1b862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb9a885102a std::function<>::operator()() at ??:0
@ 0x7fb9a88497e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fb9a88447f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fb9a8842927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fb9a8880089 handle_request at ??:0
@ 0x7fb9a888276b worker_thread at ??:0
@ 0x7fb9a2b936db start_thread at ??:0
@ 0x7fb99b9d171f clone at ??:0
W20240502 14:14:18.860656 5361 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: recv got EOF from 127.4.190.194:44571 (error 108)
W20240502 14:14:18.877130 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.886358 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.899576 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.908737 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.928050 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.942493 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.973237 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:18.987690 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.024004 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.043675 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.085562 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.108098 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.115254 4979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:19.116673 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:14:19.158128 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.183303 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.232663 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.258162 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.315958 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.347326 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
I20240502 14:14:19.394971 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:19.415455 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.449904 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.511981 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.515831 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.517637 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.519397 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.526930 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.539022 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.546859 5003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.547461 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.554450 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.557404 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.565690 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.577975 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.599828 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.607147 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.611536 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.621484 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.630580 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.644829 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.665124 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.669287 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.672111 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.705758 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.729167 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.732525 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.774168 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.791518 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.798946 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.800668 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.833824 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.846881 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.854311 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.873374 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.880141 5361 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:19.923632 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.927892 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.956775 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.967609 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
W20240502 14:14:19.969025 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
I20240502 14:14:19.999290 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:20.022859 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:20.040176 5534 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.040385 5534 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:20.041823 5534 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:20.055619 5245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:20.056543 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
I20240502 14:14:20.058836 5527 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.059015 5527 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
W20240502 14:14:20.058651 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.059835 5527 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
I20240502 14:14:20.060531 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 2 candidate_status { last_received { term: 1 index: 453 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128" is_pre_election: true
I20240502 14:14:20.060748 5307 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 1.
I20240502 14:14:20.061198 4977 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 7f06ca17ec1e4032baf8ca7598689037; no voters:
I20240502 14:14:20.061600 5527 raft_consensus.cc:2781] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:14:20.061714 5527 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.061815 5527 raft_consensus.cc:3037] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:14:20.062659 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53922: Illegal state: replica 7f06ca17ec1e4032baf8ca7598689037 is not leader of this config: current role FOLLOWER
I20240502 14:14:20.063539 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "32ace0a8418742349cdbbaaca6fbd128" candidate_term: 2 candidate_status { last_received { term: 1 index: 453 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037" is_pre_election: true
W20240502 14:14:20.063943 4979 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.064393 5534 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.064527 5534 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
W20240502 14:14:20.065052 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:20.065035 5527 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:20.065172 5534 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
I20240502 14:14:20.065740 5044 raft_consensus.cc:2379] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32ace0a8418742349cdbbaaca6fbd128 in current term 2: Already voted for candidate 7f06ca17ec1e4032baf8ca7598689037 in this term.
I20240502 14:14:20.065876 5043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "32ace0a8418742349cdbbaaca6fbd128" candidate_term: 2 candidate_status { last_received { term: 1 index: 462 } } ignore_live_leader: false dest_uuid: "7f06ca17ec1e4032baf8ca7598689037" is_pre_election: true
I20240502 14:14:20.066679 5244 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
W20240502 14:14:20.067276 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.067426 5534 raft_consensus.cc:3037] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:20.067778 5527 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: Requested vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459)
W20240502 14:14:20.067967 4979 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.067425 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:20.068439 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc78b71a424b069d2c823e9be1bdd7" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 2 candidate_status { last_received { term: 1 index: 453 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128"
I20240502 14:14:20.069108 5539 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:14:20.069962 5534 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:14:20.071540 5527 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.071695 5527 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:20.071991 5307 raft_consensus.cc:2454] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 2.
I20240502 14:14:20.072382 5527 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
I20240502 14:14:20.072728 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 2 candidate_status { last_received { term: 1 index: 462 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128" is_pre_election: true
I20240502 14:14:20.072903 5307 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 1.
I20240502 14:14:20.072891 4977 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 7f06ca17ec1e4032baf8ca7598689037; no voters: 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:20.073319 5527 raft_consensus.cc:2781] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:20.073365 4977 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 7f06ca17ec1e4032baf8ca7598689037; no voters:
W20240502 14:14:20.074188 4979 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.074671 5527 raft_consensus.cc:686] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [term 2 LEADER]: Becoming Leader. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Running, Role: LEADER
I20240502 14:14:20.074966 5527 consensus_queue.cc:237] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 453, Committed index: 453, Last appended: 1.453, Last appended by leader: 453, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:20.075155 5540 raft_consensus.cc:2781] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:14:20.075243 5540 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83db009b0a0a4b189a7bc71211a72921)
I20240502 14:14:20.075372 5540 raft_consensus.cc:3037] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:20.079241 5540 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:20.080864 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ccdc3ce29154921b753cc603d8ded3e" candidate_uuid: "7f06ca17ec1e4032baf8ca7598689037" candidate_term: 2 candidate_status { last_received { term: 1 index: 462 } } ignore_live_leader: false dest_uuid: "32ace0a8418742349cdbbaaca6fbd128"
I20240502 14:14:20.081071 5307 raft_consensus.cc:3037] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:14:20.082480 4979 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:20.082976 5540 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: Requested vote from peers 32ace0a8418742349cdbbaaca6fbd128 (127.4.190.195:43459), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
I20240502 14:14:20.084267 5307 raft_consensus.cc:2454] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f06ca17ec1e4032baf8ca7598689037 in term 2.
I20240502 14:14:20.085425 4902 catalog_manager.cc:5484] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 reported cstate change: term changed from 1 to 2, leader changed from 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194) to 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193). New cstate: current_term: 2 leader_uuid: "7f06ca17ec1e4032baf8ca7598689037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:20.086261 4977 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128, 7f06ca17ec1e4032baf8ca7598689037; no voters: 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:20.086547 5527 raft_consensus.cc:2781] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:20.086691 5527 raft_consensus.cc:686] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [term 2 LEADER]: Becoming Leader. State: Replica: 7f06ca17ec1e4032baf8ca7598689037, State: Running, Role: LEADER
I20240502 14:14:20.086980 5527 consensus_queue.cc:237] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 1.462, Last appended by leader: 462, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:20.094354 4902 catalog_manager.cc:5484] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 reported cstate change: term changed from 1 to 2, leader changed from 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194) to 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193). New cstate: current_term: 2 leader_uuid: "7f06ca17ec1e4032baf8ca7598689037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } health_report { overall_health: HEALTHY } } }
W20240502 14:14:20.110019 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:20.122956 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:20.140203 5307 raft_consensus.cc:1261] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Refusing update from remote peer 7f06ca17ec1e4032baf8ca7598689037: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 2 index: 455. (index mismatch)
I20240502 14:14:20.140925 5527 consensus_queue.cc:1035] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
W20240502 14:14:20.141345 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:20.163576 5307 raft_consensus.cc:1261] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Refusing update from remote peer 7f06ca17ec1e4032baf8ca7598689037: Log matching property violated. Preceding OpId in replica: term: 1 index: 462. Preceding OpId from leader: term: 2 index: 464. (index mismatch)
W20240502 14:14:20.164330 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:20.164435 5527 consensus_queue.cc:1035] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I20240502 14:14:20.591192 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:20.704377 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
W20240502 14:14:21.465736 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:14:21.723273 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:21.984553 5246 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.626s user 0.002s sys 0.000s
I20240502 14:14:21.990550 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 1.399s user 0.551s sys 0.077s Metrics: {"bytes_written":150208,"cfile_init":1,"dirs.queue_time_us":6376,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":7847,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2128,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7929,"thread_start_us":6529,"threads_started":1}
I20240502 14:14:21.992692 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014686
W20240502 14:14:22.202023 4980 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.790s user 0.000s sys 0.000s
I20240502 14:14:22.203236 5554 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53922 (ReqId={client: 56c84f4e659c43d78e50b71ec5301770, seq_no=1504, attempt_no=0}) took 1009 ms. Trace:
I20240502 14:14:22.203475 5554 rpcz_store.cc:276] 0502 14:14:21.193359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:14:21.193416 (+ 57us) service_pool.cc:225] Handling call
0502 14:14:22.203217 (+1009801us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 14:14:21.193963 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4ccdc3ce29154921b753cc603d8ded3e
0502 14:14:21.194080 (+ 117us) write_op.cc:424] Acquiring schema lock in shared mode
0502 14:14:21.194084 (+ 4us) write_op.cc:427] Acquired schema lock
0502 14:14:21.194086 (+ 2us) tablet.cc:660] Decoding operations
0502 14:14:21.202349 (+ 8263us) write_op.cc:612] Acquiring the partition lock for write op
0502 14:14:21.202366 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0502 14:14:21.202373 (+ 7us) tablet.cc:683] Acquiring locks for 1525 operations
0502 14:14:21.291334 (+ 88961us) tablet.cc:699] Row locks acquired
0502 14:14:21.291361 (+ 27us) write_op.cc:260] PREPARE: finished
0502 14:14:21.291448 (+ 87us) write_op.cc:270] Start()
0502 14:14:21.291489 (+ 41us) write_op.cc:276] Timestamp: P: 1714659261291404 usec, L: 0
0502 14:14:21.291493 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 14:14:21.291773 (+ 280us) log.cc:842] Serialized 17125 byte log entry
0502 14:14:21.303476 (+ 11703us) op_driver.cc:444] REPLICATION: finished
0502 14:14:21.307716 (+ 4240us) write_op.cc:301] APPLY: starting
0502 14:14:21.307737 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0502 14:14:21.436005 (+128268us) tablet.cc:1368] finished BulkCheckPresence
0502 14:14:21.436037 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 14:14:22.198538 (+762501us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 14:14:22.199386 (+ 848us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=1526,key_file_lookups=1001,delta_file_lookups=0,mrs_lookups=3050
0502 14:14:22.199395 (+ 9us) write_op.cc:312] APPLY: finished
0502 14:14:22.201882 (+ 2487us) log.cc:842] Serialized 15176 byte log entry
0502 14:14:22.201979 (+ 97us) write_op.cc:481] Releasing partition, row and schema locks
0502 14:14:22.203017 (+ 1038us) write_op.cc:446] Released schema lock
0502 14:14:22.203117 (+ 100us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4172,"cfile_cache_hit":2804,"cfile_cache_hit_bytes":50706224,"cfile_cache_miss":8,"cfile_cache_miss_bytes":68001,"cfile_init":1,"lbm_read_time_us":375,"lbm_reads_lt_1ms":12,"num_ops":1525,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":23650,"prepare.run_wall_time_us":98906,"raft.queue_time_us":111,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":180,"replication_time_us":11955,"spinlock_wait_cycles":384}]]}
I20240502 14:14:22.205961 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 1.500s user 0.463s sys 0.082s Metrics: {"bytes_written":152102,"cfile_init":1,"dirs.queue_time_us":6633,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":4240,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1977,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8064,"thread_start_us":6788,"threads_started":1}
I20240502 14:14:22.212538 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 136120 bytes on disk
I20240502 14:14:22.213289 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 14:14:22.214141 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014818
I20240502 14:14:22.699481 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:22.701329 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:14:22.771963 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:14:22.893087 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.900s user 0.767s sys 0.003s Metrics: {"bytes_written":293375,"cfile_cache_hit":14,"cfile_cache_hit_bytes":187168,"cfile_cache_miss":54,"cfile_cache_miss_bytes":870810,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":420,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":3065,"drs_written":1,"lbm_read_time_us":3107,"lbm_reads_lt_1ms":86,"lbm_write_time_us":3294,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":502583,"rows_written":15856,"thread_start_us":331,"threads_started":1}
I20240502 14:14:22.894002 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 293067 bytes on disk
I20240502 14:14:22.894623 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 14:14:23.009092 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.795s user 0.607s sys 0.018s Metrics: {"bytes_written":296352,"cfile_cache_hit":15,"cfile_cache_hit_bytes":189115,"cfile_cache_miss":57,"cfile_cache_miss_bytes":878881,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":2035,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":2441,"drs_written":1,"lbm_read_time_us":3849,"lbm_reads_lt_1ms":89,"lbm_write_time_us":4174,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":508716,"rows_written":16026,"thread_start_us":1914,"threads_started":1}
I20240502 14:14:23.010186 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 296438 bytes on disk
I20240502 14:14:23.011066 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":260,"lbm_reads_lt_1ms":4}
W20240502 14:14:23.724520 5403 debug-util.cc:398] Leaking SignalData structure 0x60300085b880 after lost signal to thread 5554
W20240502 14:14:23.823089 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:14:24.182308 4979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:14:25.191848 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:14:25.330379 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:14:25.502487 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:25.655869 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:25.768527 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:26.261118 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.605s user 0.418s sys 0.015s Metrics: {"bytes_written":144650,"cfile_init":1,"dirs.queue_time_us":6383,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":58485,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2770,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7885,"thread_start_us":6503,"threads_started":1}
I20240502 14:14:26.262197 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014709
I20240502 14:14:26.371932 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:26.408699 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:14:27.020651 4859 linked_list-test-util.h:417] Sample at HT timestamp: 7023244357639036928 Inserted count: 37800
W20240502 14:14:27.046859 4980 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.586s user 0.001s sys 0.000s
I20240502 14:14:27.077575 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 1.308s user 0.510s sys 0.006s Metrics: {"bytes_written":144736,"cfile_init":1,"dirs.queue_time_us":365,"dirs.run_cpu_time_us":540,"dirs.run_wall_time_us":90092,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2194,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7885,"thread_start_us":248,"threads_started":1}
I20240502 14:14:27.079087 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 133831 bytes on disk
I20240502 14:14:27.079787 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 14:14:27.080693 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014707
W20240502 14:14:27.635915 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:14:28.112121 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:14:28.288554 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.026s user 1.004s sys 0.054s Metrics: {"bytes_written":3826,"cfile_cache_hit":11,"cfile_cache_hit_bytes":47150,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1559974,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":83488,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":76314,"drs_written":1,"lbm_read_time_us":10727,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":106,"lbm_write_time_us":4609,"lbm_writes_lt_1ms":66,"num_input_rowsets":2,"peak_mem_usage":774952,"reinsert_count":0,"rows_written":23741,"thread_start_us":84773,"threads_started":8,"update_count":568}
I20240502 14:14:28.290355 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=0.010042
W20240502 14:14:28.439488 5391 log.cc:924] Time spent T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20240502 14:14:28.439488 5544 log.cc:924] Time spent T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240502 14:14:28.439730 5417 log.cc:924] Time spent T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20240502 14:14:28.439807 5550 log.cc:924] Time spent T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20240502 14:14:28.447499 5553 log.cc:924] Time spent T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20240502 14:14:28.448830 5542 log.cc:924] Time spent T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240502 14:14:28.468981 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 1.388s user 1.025s sys 0.021s Metrics: {"bytes_written":439526,"cfile_cache_hit":14,"cfile_cache_hit_bytes":277898,"cfile_cache_miss":72,"cfile_cache_miss_bytes":1339438,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":23734,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":37956,"drs_written":1,"lbm_read_time_us":4417,"lbm_reads_lt_1ms":100,"lbm_write_time_us":7165,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":62,"mutex_wait_us":1040,"num_input_rowsets":2,"peak_mem_usage":777606,"rows_written":23911,"thread_start_us":21133,"threads_started":7}
I20240502 14:14:28.470037 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 447973 bytes on disk
I20240502 14:14:28.477157 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:14:28.550885 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.253s user 0.082s sys 0.012s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5558,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":543,"lbm_reads_lt_1ms":24,"lbm_write_time_us":43729,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"update_count":568}
I20240502 14:14:28.551815 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 439592 bytes on disk
I20240502 14:14:28.552428 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 14:14:29.081691 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:29.295614 4979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:14:29.296878 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:14:29.912451 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:30.009433 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:14:30.557747 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab): perf score=1.000000
I20240502 14:14:30.733007 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(3fcc78b71a424b069d2c823e9be1bdd7): perf score=1.000000
W20240502 14:14:30.806311 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:14:31.067427 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab) complete. Timing: real 0.509s user 0.408s sys 0.067s Metrics: {"bytes_written":143437,"cfile_init":1,"dirs.queue_time_us":6437,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":9459,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1886,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7673,"thread_start_us":6564,"threads_started":1}
I20240502 14:14:31.068522 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(3fcc78b71a424b069d2c823e9be1bdd7): perf score=1.000000
I20240502 14:14:31.338371 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(3fcc78b71a424b069d2c823e9be1bdd7) complete. Timing: real 0.605s user 0.445s sys 0.013s Metrics: {"bytes_written":142873,"cfile_init":1,"dirs.queue_time_us":10810,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1217,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2095,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7631,"thread_start_us":10680,"threads_started":1}
I20240502 14:14:31.339483 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(3fcc78b71a424b069d2c823e9be1bdd7): 143881 bytes on disk
I20240502 14:14:31.340142 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(3fcc78b71a424b069d2c823e9be1bdd7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 14:14:31.340837 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab): perf score=1.000000
W20240502 14:14:31.784744 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:14:31.883062 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(3fcc78b71a424b069d2c823e9be1bdd7) complete. Timing: real 0.814s user 0.461s sys 0.058s Metrics: {"bytes_written":144434,"cfile_init":1,"dirs.queue_time_us":6325,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":11336,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2354,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7720,"thread_start_us":6463,"threads_started":1}
I20240502 14:14:31.884902 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(3fcc78b71a424b069d2c823e9be1bdd7): 144876 bytes on disk
I20240502 14:14:31.894021 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(3fcc78b71a424b069d2c823e9be1bdd7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 14:14:31.895105 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:32.029563 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab) complete. Timing: real 0.688s user 0.546s sys 0.061s Metrics: {"bytes_written":1604,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12831,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1046,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2250,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7885,"thread_start_us":12719,"threads_started":1,"update_count":222}
I20240502 14:14:32.030864 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.000000
I20240502 14:14:32.107481 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:32.566064 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.671s user 0.467s sys 0.064s Metrics: {"bytes_written":148733,"cfile_init":1,"dirs.queue_time_us":6339,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":2083,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2681,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8103,"thread_start_us":6478,"threads_started":1}
I20240502 14:14:32.572563 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab): 144807 bytes on disk
I20240502 14:14:32.573174 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 14:14:32.575564 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014680
I20240502 14:14:32.658941 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: FlushMRSOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.627s user 0.509s sys 0.048s Metrics: {"bytes_written":149428,"cfile_init":1,"dirs.queue_time_us":6310,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":8584,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1962,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8133,"thread_start_us":6457,"threads_started":1}
I20240502 14:14:32.667845 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab): 147060 bytes on disk
I20240502 14:14:32.670547 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
I20240502 14:14:32.671525 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=1.014573
W20240502 14:14:32.725613 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:14:33.302541 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:14:33.403191 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:34.061305 5602 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53922 (ReqId={client: 56c84f4e659c43d78e50b71ec5301770, seq_no=2965, attempt_no=0}) took 1455 ms. Trace:
I20240502 14:14:34.061524 5602 rpcz_store.cc:276] 0502 14:14:32.605713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:14:32.605781 (+ 68us) service_pool.cc:225] Handling call
0502 14:14:34.061284 (+1455503us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 14:14:32.606350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4ccdc3ce29154921b753cc603d8ded3e
0502 14:14:32.606478 (+ 128us) write_op.cc:424] Acquiring schema lock in shared mode
0502 14:14:32.606484 (+ 6us) write_op.cc:427] Acquired schema lock
0502 14:14:32.606486 (+ 2us) tablet.cc:660] Decoding operations
0502 14:14:32.626773 (+ 20287us) write_op.cc:612] Acquiring the partition lock for write op
0502 14:14:32.626790 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0502 14:14:32.626796 (+ 6us) tablet.cc:683] Acquiring locks for 1021 operations
0502 14:14:32.653161 (+ 26365us) tablet.cc:699] Row locks acquired
0502 14:14:32.653167 (+ 6us) write_op.cc:260] PREPARE: finished
0502 14:14:32.653243 (+ 76us) write_op.cc:270] Start()
0502 14:14:32.653285 (+ 42us) write_op.cc:276] Timestamp: P: 1714659272653212 usec, L: 0
0502 14:14:32.653291 (+ 6us) op_driver.cc:331] REPLICATION: starting
0502 14:14:32.653563 (+ 272us) log.cc:842] Serialized 11576 byte log entry
0502 14:14:32.664777 (+ 11214us) op_driver.cc:444] REPLICATION: finished
0502 14:14:32.664877 (+ 100us) write_op.cc:301] APPLY: starting
0502 14:14:32.664892 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0502 14:14:33.284612 (+619720us) tablet.cc:1368] finished BulkCheckPresence
0502 14:14:33.284641 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 14:14:34.054881 (+770240us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 14:14:34.055587 (+ 706us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=2549,key_file_lookups=1014,delta_file_lookups=0,mrs_lookups=514
0502 14:14:34.055596 (+ 9us) write_op.cc:312] APPLY: finished
0502 14:14:34.059867 (+ 4271us) log.cc:842] Serialized 7160 byte log entry
0502 14:14:34.060076 (+ 209us) write_op.cc:481] Releasing partition, row and schema locks
0502 14:14:34.061103 (+ 1027us) write_op.cc:446] Released schema lock
0502 14:14:34.061174 (+ 71us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"cfile_cache_hit":2442,"cfile_cache_hit_bytes":50095067,"cfile_cache_miss":2,"cfile_cache_miss_bytes":47770,"lbm_read_time_us":99,"lbm_reads_lt_1ms":2,"num_ops":1021,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":29111,"prepare.run_wall_time_us":55173,"raft.queue_time_us":5866,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":378,"replication_time_us":11459,"spinlock_wait_cycles":6016}]]}
W20240502 14:14:34.328204 4979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 14:14:34.530306 4979 consensus_peers.cc:485] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:14:34.736143 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.064s user 1.429s sys 0.062s Metrics: {"bytes_written":3452,"cfile_cache_hit":11,"cfile_cache_hit_bytes":48722,"cfile_cache_miss":91,"cfile_cache_miss_bytes":2147740,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":17111,"dirs.run_cpu_time_us":1447,"dirs.run_wall_time_us":17775,"drs_written":1,"lbm_read_time_us":5607,"lbm_reads_lt_1ms":123,"lbm_write_time_us":6651,"lbm_writes_lt_1ms":80,"num_input_rowsets":2,"peak_mem_usage":1054191,"reinsert_count":0,"rows_written":32044,"thread_start_us":61891,"threads_started":4,"update_count":507}
I20240502 14:14:34.737640 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 601095 bytes on disk
I20240502 14:14:34.740849 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 14:14:34.741748 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling MajorDeltaCompactionOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab): perf score=0.012681
I20240502 14:14:34.803809 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: MajorDeltaCompactionOp(6ba0a6a6ce8945e4bfba8c6c31ef7dab) complete. Timing: real 0.056s user 0.049s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2237,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":100,"lbm_read_time_us":358,"lbm_reads_lt_1ms":20,"lbm_write_time_us":278,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":222}
I20240502 14:14:34.804675 5093 maintenance_manager.cc:419] P 7f06ca17ec1e4032baf8ca7598689037: Scheduling MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=0.006745
I20240502 14:14:35.003166 4980 maintenance_manager.cc:643] P 7f06ca17ec1e4032baf8ca7598689037: MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.197s user 0.173s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5178,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":73,"lbm_read_time_us":455,"lbm_reads_lt_1ms":20,"lbm_write_time_us":301,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":507}
I20240502 14:14:35.047360 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: CompactRowSetsOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 2.472s user 1.371s sys 0.195s Metrics: {"bytes_written":2192,"cfile_cache_hit":12,"cfile_cache_hit_bytes":48587,"cfile_cache_miss":92,"cfile_cache_miss_bytes":2133643,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":21372,"dirs.run_cpu_time_us":2283,"dirs.run_wall_time_us":33963,"drs_written":1,"lbm_read_time_us":4775,"lbm_reads_lt_1ms":128,"lbm_write_time_us":18259,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":79,"mutex_wait_us":300,"num_input_rowsets":2,"peak_mem_usage":1051026,"reinsert_count":0,"rows_written":31844,"thread_start_us":304167,"threads_started":4,"update_count":307}
I20240502 14:14:35.052322 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e): 598901 bytes on disk
I20240502 14:14:35.053088 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: UndoDeltaBlockGCOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 14:14:35.054414 5356 maintenance_manager.cc:419] P 32ace0a8418742349cdbbaaca6fbd128: Scheduling MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e): perf score=0.004313
I20240502 14:14:35.258561 5246 maintenance_manager.cc:643] P 32ace0a8418742349cdbbaaca6fbd128: MajorDeltaCompactionOp(4ccdc3ce29154921b753cc603d8ded3e) complete. Timing: real 0.204s user 0.138s sys 0.012s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3342,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":74,"lbm_read_time_us":385,"lbm_reads_lt_1ms":20,"lbm_write_time_us":358,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":307}
W20240502 14:14:35.357264 4979 consensus_peers.cc:485] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:14:35.602397 4979 consensus_peers.cc:485] T 3fcc78b71a424b069d2c823e9be1bdd7 P 7f06ca17ec1e4032baf8ca7598689037 -> Peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Couldn't send request to peer 83db009b0a0a4b189a7bc71211a72921. Status: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:14:35.893339 5403 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:36.526162 5640 consensus_queue.cc:786] T 4ccdc3ce29154921b753cc603d8ded3e P 7f06ca17ec1e4032baf8ca7598689037 [LEADER]: Peer 83db009b0a0a4b189a7bc71211a72921 is lagging by at least 12 ops behind the committed index
F20240502 14:14:36.551899 5403 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fc652cc6dcd google::LogMessage::Fail() at ??:0
@ 0x7fc652ccab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc652cc67cc google::LogMessage::Flush() at ??:0
@ 0x7fc652cc7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc6558efc80 kudu::operator-() at ??:0
@ 0x7fc66257014a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fc66253a5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fc66253a336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fc6624e72d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fc664fb4e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fc664fc39fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fc664fc3862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc65e9f902a std::function<>::operator()() at ??:0
@ 0x7fc65e9f17e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fc65e9ec7f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fc65e9ea927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fc65ea28089 handle_request at ??:0
@ 0x7fc65ea2a76b worker_thread at ??:0
@ 0x7fc658d3b6db start_thread at ??:0
@ 0x7fc651b7971f clone at ??:0
W20240502 14:14:36.844017 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
W20240502 14:14:36.858453 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:36.885860 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:36.910066 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:36.918102 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:36.963937 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.025748 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.027074 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.027997 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.032023 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.050971 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.053306 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.067376 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.079874 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.121395 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.124624 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.132841 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.150592 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.176504 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.186219 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.201910 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.230084 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.268342 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.268435 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.296770 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.328660 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.370182 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.420114 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.424626 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.456925 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.467360 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.550452 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.562390 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.612823 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.654249 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.671841 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.682971 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.770143 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.784822 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.837571 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.853996 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:37.895820 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:37.904008 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.023681 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:38.025247 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:38.034817 5675 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:38.034982 5675 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:38.036015 5675 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:38.036958 5245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:14:38.038189 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:38.038415 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:38.038519 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:38.038785 5675 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:38.040777 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:38.049925 5675 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:38.050094 5675 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:38.050655 5675 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:38.051875 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:38.052109 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:38.052265 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:38.052532 5675 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:14:38.055912 5675 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:38.056083 5675 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:38.056655 5675 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:38.057898 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:38.058073 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:38.058216 5244 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:38.058456 5675 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:38.097980 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.097995 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.161463 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.233992 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.309687 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.388315 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.390478 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.467289 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.469045 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.633214 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.704170 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.716732 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.720078 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.812955 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:38.874655 5361 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:38.896870 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.004245 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:39.052616 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:39.085942 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.088141 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.192862 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.399189 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.426061 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.434850 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.490679 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.611038 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.644639 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.830351 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:39.865296 5679 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:39.865509 5679 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:39.866259 5679 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:39.868218 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:39.868671 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:39.868820 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:39.869136 5679 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:39.876914 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:39.934543 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.944031 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:39.952555 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:40.012724 5679 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:40.012941 5679 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:40.013648 5679 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:40.015306 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:40.015830 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:40.015957 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:40.016227 5679 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:40.057816 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:40.075361 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:40.292809 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:40.300246 5679 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:40.300442 5679 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:40.301218 5679 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:40.302842 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:40.303025 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:40.303195 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:40.303531 5679 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:14:40.347119 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.408831 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.439651 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.540378 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.797590 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.857843 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.878747 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:40.921075 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:40.969091 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.061033 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:41.094318 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:41.336247 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.436331 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.468533 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.494360 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.620613 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.730576 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:41.882568 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:41.910907 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.032385 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.054790 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:42.097965 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:42.214531 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.345333 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:42.455837 5683 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:42.456076 5683 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:42.456916 5683 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:42.459055 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:42.459360 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:42.459523 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:42.459831 5683 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:14:42.526188 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:42.561079 5683 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:42.561259 5683 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:42.561901 5683 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:42.563288 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:42.563537 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:42.563680 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:42.563957 5683 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:42.667958 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.670924 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:42.678721 5683 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:42.678910 5683 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:42.679713 5683 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:42.681126 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:42.681354 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:42.681448 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:42.681708 5683 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:42.698916 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.845546 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:42.886106 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:43.098996 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:43.175846 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.218348 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.328147 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.357128 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.515266 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.864776 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:43.905777 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:43.912714 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.026549 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.055483 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:44.128687 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:44.224459 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.591238 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.640691 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.768270 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.782897 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:44.908537 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:44.963590 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:45.133448 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:45.143700 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:45.346226 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:45.529610 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:45.534914 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:45.738215 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:45.923727 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:46.041832 5687 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:46.042011 5687 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:46.042744 5687 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:46.043879 5244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240502 14:14:46.045192 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:46.045545 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:46.045655 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:46.045933 5687 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:14:46.129191 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:46.139634 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:46.209960 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:46.216173 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:46.324712 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:46.495066 5687 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:46.495326 5687 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:46.496344 5687 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:46.497686 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:46.498140 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:46.498273 5244 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:46.498587 5687 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:14:46.553903 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:46.623059 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:46.894798 5687 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:46.895095 5687 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:46.896138 5687 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:46.897421 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:46.897728 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:46.897907 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:46.898200 5687 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:46.974731 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:47.015638 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:47.155833 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:47.161841 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:47.189256 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:47.197309 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:47.403101 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:47.842651 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:47.918723 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:48.029686 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:48.070035 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:48.078833 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:48.194274 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:48.291491 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:48.745585 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:48.798003 5691 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:48.798290 5691 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:48.799019 5691 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:48.801265 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:48.801483 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:48.801668 5244 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:48.801987 5691 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:48.944505 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:48.984604 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:49.097997 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:49.195056 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:49.217178 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:49.295195 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:49.687978 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:49.818130 5695 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:49.818418 5695 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:49.819321 5695 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:49.825220 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:49.825614 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:49.825755 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:49.826038 5695 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:14:49.888197 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:49.926962 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.172159 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.211405 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:50.226536 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:14:50.240769 5695 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:50.240969 5695 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:50.241670 5695 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:50.244585 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:50.248550 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:50.248679 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:50.248955 5695 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:50.252262 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.660902 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.746716 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.870487 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:50.916934 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:51.167608 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:51.248844 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:51.320586 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:51.675683 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:51.760140 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:51.887952 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:51.946244 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:52.188730 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:52.281026 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:52.325985 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:52.717200 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:52.809404 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:52.940655 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:53.009267 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:53.250583 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:53.300567 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:53.324654 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:53.796240 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:53.893602 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:54.031250 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:54.108096 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:54.344033 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:54.349270 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:54.402441 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:54.910629 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:55.006719 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:55.154867 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:55.242311 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:55.346303 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:55.434091 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:55.486104 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:56.064018 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:56.166702 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:56.315451 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:56.367293 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:56.408865 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:56.453759 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:56.650951 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:57.248867 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:57.354903 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:57.367926 5699 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:57.368113 5699 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:14:57.368842 5699 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:14:57.369743 5245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 14:14:57.370029 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:57.370960 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:57.371282 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:14:57.371420 5244 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:57.371683 5699 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:57.517375 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:57.528139 5699 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:14:57.528353 5699 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:14:57.529134 5699 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:14:57.530858 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:57.531668 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:14:57.531785 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:14:57.532104 5699 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:14:57.558320 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:57.615767 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:57.857715 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:58.415472 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:58.476084 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:58.583102 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:58.650595 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:14:58.748914 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:58.860026 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:59.099733 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:14:59.433281 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:14:59.706107 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:14:59.734444 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:14:59.847859 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:00.016482 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:00.134915 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:00.382256 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:00.434579 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:00.715553 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:01.000095 5704 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:01.000284 5704 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:01.000927 5704 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:01.003757 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:01.003978 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:01.004135 5244 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:01.004429 5704 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:01.042097 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:01.148407 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:01.329437 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:01.442616 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:01.456892 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:01.700349 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:01.771639 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:02.366883 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:02.483120 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:02.498085 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:02.670605 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:02.791457 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:02.813068 5708 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:02.813257 5708 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:02.813947 5708 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:02.814965 5244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240502 14:15:02.816217 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:02.816642 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:02.816788 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:02.817107 5708 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:02.819712 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:03.045596 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:03.550793 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:03.732208 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:03.860759 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:03.948527 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:04.049521 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:04.183991 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:04.435724 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:04.562937 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:04.994352 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:05.144408 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:05.269455 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:05.468978 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:05.564838 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:05.604058 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:05.861672 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:06.085271 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:06.562618 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:06.584977 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:06.715503 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:06.924065 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:07.061404 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:07.171094 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:07.323735 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:07.564728 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:08.070087 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:08.198419 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:08.303035 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:08.409689 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:08.555732 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:08.569734 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:08.822861 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:09.315742 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:09.565992 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:09.580364 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:09.711820 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:09.934933 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:10.084708 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:10.345304 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:10.358989 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:10.638159 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:11.141108 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:11.265566 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:11.388422 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:11.493058 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:11.647838 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:11.652951 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:11.932672 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:11.957360 5712 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:11.957525 5712 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:11.958283 5712 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:11.959609 5244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:15:11.960929 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:11.961148 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:11.961249 5244 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:11.961519 5712 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:15:12.213069 5712 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:12.213268 5712 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:15:12.213958 5712 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:15:12.215301 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:12.215514 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:12.215662 5245 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:12.215951 5712 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:12.449031 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:12.638549 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:12.730234 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:12.855876 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:13.085608 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:13.254031 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:13.526564 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:13.537695 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:13.661322 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:14.357499 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:14.476821 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:14.617594 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:14.680711 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:14.719942 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:14.891261 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:15.179661 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:15.678465 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:15.722566 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:15.843696 5716 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:15.843881 5716 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:15.844655 5716 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:15.846688 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:15.846855 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:15.846976 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:15.847215 5716 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:16.016644 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:16.138219 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:16.392084 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:16.562805 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:16.689988 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:16.845084 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:16.860342 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:17.712455 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:17.757979 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:15:17.779242 5720 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:17.779448 5720 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:15:17.780126 5720 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:15:17.780900 5245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240502 14:15:17.782091 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:17.782596 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:17.782701 5244 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:17.782948 5720 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:17.832795 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:17.984794 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:18.097249 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:18.271477 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:18.368399 5724 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:18.368568 5724 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:18.369303 5724 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:18.371615 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:18.371852 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:18.371970 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:18.372195 5724 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:18.571712 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:18.775616 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:19.132176 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:19.437888 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:19.563462 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:19.816704 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:19.839891 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:20.010366 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:20.135499 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:20.314349 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:20.827987 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:21.193964 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:21.202427 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:21.326205 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:21.463937 5728 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:21.464138 5728 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:21.464866 5728 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:21.467278 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:21.467428 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:21.467593 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:21.467903 5728 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:21.618916 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:21.789530 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:21.853801 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:22.101936 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:22.194249 5733 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:22.194455 5733 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:15:22.195142 5733 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:15:22.197566 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:22.197708 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:22.197803 5244 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:22.198113 5733 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:22.216431 5361 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:22.893765 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:23.006798 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:23.137055 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:23.303215 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:23.432180 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:23.610514 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:23.909488 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:23.919410 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:24.352175 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:24.845055 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:24.934543 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:24.978592 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:25.280964 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:25.462479 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:25.465058 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:25.776537 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:25.971958 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:15:26.229193 5737 raft_consensus.cc:482] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:26.229382 5737 raft_consensus.cc:504] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } }
I20240502 14:15:26.230096 5737 leader_election.cc:288] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571), 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177)
W20240502 14:15:26.230962 5245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240502 14:15:26.232215 5245 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:26.232409 5244 leader_election.cc:334] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:26.232555 5244 leader_election.cc:302] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:26.232812 5737 raft_consensus.cc:2726] T 4ccdc3ce29154921b753cc603d8ded3e P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:26.534456 5360 meta_cache.cc:302] tablet 6ba0a6a6ce8945e4bfba8c6c31ef7dab: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:26.719659 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:26.974328 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:27.038450 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:27.162468 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:27.173468 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:27.671953 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:27.676729 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:27.995769 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:28.000725 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:28.631876 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:28.765058 5361 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:28.961200 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:29.004375 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:29.090441 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:29.598706 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:29.925372 5361 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
W20240502 14:15:29.933862 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:30.035698 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:30.579990 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:30.919723 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:31.042793 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:31.044085 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:31.054144 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 14:15:31.227278 5742 raft_consensus.cc:482] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:31.227524 5742 raft_consensus.cc:504] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:31.228281 5742 leader_election.cc:288] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:31.231086 5244 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:31.231261 5245 leader_election.cc:334] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:31.231459 5245 leader_election.cc:302] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:31.231796 5742 raft_consensus.cc:2726] T 6ba0a6a6ce8945e4bfba8c6c31ef7dab P 32ace0a8418742349cdbbaaca6fbd128 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:31.238431 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:31.568400 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:31.909654 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:32.072350 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:32.244678 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:32.562548 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:32.916260 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:33.029018 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:33.031199 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:33.117822 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:33.252467 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:33.563057 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:33.928107 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:34.163894 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:34.264955 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:34.581318 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:34.949177 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:35.056248 5269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:35.056924 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:35.163579 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:35.288636 5360 meta_cache.cc:302] tablet 3fcc78b71a424b069d2c823e9be1bdd7: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
I20240502 14:15:35.313051 5746 raft_consensus.cc:482] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f06ca17ec1e4032baf8ca7598689037)
I20240502 14:15:35.313223 5746 raft_consensus.cc:504] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f06ca17ec1e4032baf8ca7598689037" member_type: VOTER last_known_addr { host: "127.4.190.193" port: 34177 } } peers { permanent_uuid: "83db009b0a0a4b189a7bc71211a72921" member_type: VOTER last_known_addr { host: "127.4.190.194" port: 44571 } } peers { permanent_uuid: "32ace0a8418742349cdbbaaca6fbd128" member_type: VOTER last_known_addr { host: "127.4.190.195" port: 43459 } }
I20240502 14:15:35.313963 5746 leader_election.cc:288] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177), 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571)
W20240502 14:15:35.315109 5244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 14:15:35.316239 5244 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177): Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:35.316524 5245 leader_election.cc:334] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83db009b0a0a4b189a7bc71211a72921 (127.4.190.194:44571): Network error: Client connection negotiation failed: client connection to 127.4.190.194:44571: connect: Connection refused (error 111)
I20240502 14:15:35.316695 5245 leader_election.cc:302] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32ace0a8418742349cdbbaaca6fbd128; no voters: 7f06ca17ec1e4032baf8ca7598689037, 83db009b0a0a4b189a7bc71211a72921
I20240502 14:15:35.316998 5746 raft_consensus.cc:2726] T 3fcc78b71a424b069d2c823e9be1bdd7 P 32ace0a8418742349cdbbaaca6fbd128 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:35.605931 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:35.975912 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
I20240502 14:15:36.193701 5399 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 14:15:36.292397 5360 meta_cache.cc:302] tablet 4ccdc3ce29154921b753cc603d8ded3e: replica 7f06ca17ec1e4032baf8ca7598689037 (127.4.190.193:34177) has failed: Network error: Client connection negotiation failed: client connection to 127.4.190.193:34177: connect: Connection refused (error 111)
W20240502 14:15:36.635318 5268 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54716: Illegal state: replica 32ace0a8418742349cdbbaaca6fbd128 is not leader of this config: current role FOLLOWER
W20240502 14:15:36.666698 5360 batcher.cc:441] Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.667456 4859 client-test-util.cc:49] 9 failed ops. First 10 errors follow
I20240502 14:15:36.667651 4859 client-test-util.cc:55] Op INSERT int64 rand_key=9170206513769218050, int64 link_to=-3948404939783929854, int64 insert_ts=1714659276538832 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.667910 4859 client-test-util.cc:55] Op INSERT int64 rand_key=7708549292775243782, int64 link_to=-4991044585440346106, int64 insert_ts=1714659276539016 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668007 4859 client-test-util.cc:55] Op INSERT int64 rand_key=6247173546757980170, int64 link_to=-6033684231096762358, int64 insert_ts=1714659276539182 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668077 4859 client-test-util.cc:55] Op INSERT int64 rand_key=8640002078384193553, int64 link_to=8875249747957055505, int64 insert_ts=1714659276539550 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668157 4859 client-test-util.cc:55] Op INSERT int64 rand_key=7178485594878640149, int64 link_to=7832610102300639253, int64 insert_ts=1714659276539738 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668223 4859 client-test-util.cc:55] Op INSERT int64 rand_key=8109938380487589920, int64 link_to=3252582574453489696, int64 insert_ts=1714659276540179 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668320 4859 client-test-util.cc:55] Op INSERT int64 rand_key=6648562634470326308, int64 link_to=2209942928797073444, int64 insert_ts=1714659276540398 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668402 4859 client-test-util.cc:55] Op INSERT int64 rand_key=9041391166096539691, int64 link_to=-1327867165858660309, int64 insert_ts=1714659276540686 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
I20240502 14:15:36.668496 4859 client-test-util.cc:55] Op INSERT int64 rand_key=7579874682590986287, int64 link_to=-2370506811515076561, int64 insert_ts=1714659276540861 had status Timed out: Failed to write batch of 9 ops to tablet 3fcc78b71a424b069d2c823e9be1bdd7 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 3fcc78b71a424b069d2c823e9be1bdd7, num_ops: 9, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.4.190.193:34177 (error 108)
F20240502 14:15:36.668588 4859 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714659336 (unix time) try "date -d @1714659336" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000012fb) received by PID 4859 (TID 0x7f33297d6d80) from PID 4859; stack trace: ***
@ 0x7f33311e3980 (unknown) at ??:0
@ 0x7f332a28efb7 gsignal at ??:0
@ 0x7f332a290921 abort at ??:0
@ 0x7f332cfb7dcd google::LogMessage::Fail() at ??:0
@ 0x7f332cfbbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f332cfb77cc google::LogMessage::Flush() at ??:0
@ 0x7f332cfb8f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f333ca4defd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed8c2 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test-util.h:?
@ 0x3d6588 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8889 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7f332ee520ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f332ee46bdb testing::Test::Run() at ??:0
@ 0x7f332ee46d9d testing::TestInfo::Run() at ??:0
@ 0x7f332ee47377 testing::TestSuite::Run() at ??:0
@ 0x7f332ee4777c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f332ee5260d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f332ee46e63 testing::UnitTest::Run() at ??:0
@ 0x7f3343b313c2 RUN_ALL_TESTS() at ??:0
@ 0x7f3343b2f8d9 main at ??:0
@ 0x7f332a271bf7 __libc_start_main at ??:0
@ 0x2ebf5a _start at ??:?