[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 02:00:49.357425 396 test_util.cc:268] Using random seed: -620369328
I20240501 02:00:49.364610 396 linked_list-test.cc:100] Linked List Test Configuration:
I20240501 02:00:49.364673 396 linked_list-test.cc:101] --------------
I20240501 02:00:49.364696 396 linked_list-test.cc:102] 50 chains
I20240501 02:00:49.364717 396 linked_list-test.cc:103] 3 tablets
I20240501 02:00:49.364737 396 linked_list-test.cc:104] Mutations on
I20240501 02:00:49.364758 396 linked_list-test.cc:105] --------------
I20240501 02:00:49.364850 396 ts_itest-base.cc:112] Starting cluster with:
I20240501 02:00:49.364879 396 ts_itest-base.cc:113] --------------
I20240501 02:00:49.364890 396 ts_itest-base.cc:114] 3 tablet servers
I20240501 02:00:49.364902 396 ts_itest-base.cc:115] 3 replicas per TS
I20240501 02:00:49.364912 396 ts_itest-base.cc:116] --------------
2024-05-01T02:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:00:49Z Disabled control of system clock
I20240501 02:00:49.375485 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
/tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.62:39725
--webserver_interface=127.0.99.62
--webserver_port=0
--builtin_ntp_servers=127.0.99.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.62:39725
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240501 02:00:49.486337 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:00:49.486618 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:00:49.486665 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:00:49.489976 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:00:49.490064 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:00:49.490084 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:00:49.490109 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:00:49.494236 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.99.62:39725
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.62:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 01 May 2024 01:43:15 UTC on 1a15035a6830
build id 65
I20240501 02:00:49.495308 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:00:49.496609 408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:00:49.505702 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.506521 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.507783 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:00:49.509063 408 server_base.cc:1034] running on GCE node
I20240501 02:00:49.509690 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:00:49.510851 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:00:49.512029 408 hybrid_clock.cc:648] HybridClock initialized: now 1714528849511962 us; error 86 us; skew 500 ppm
I20240501 02:00:49.518652 408 webserver.cc:460] Webserver started at http://127.0.99.62:42445/ using document root <none> and password file <none>
I20240501 02:00:49.519358 408 fs_manager.cc:359] Metadata directory not provided
I20240501 02:00:49.519435 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:00:49.519642 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:00:49.521781 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data/instance:
uuid: "66670edd7d0b45e9a25dcdf531c5d649"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.522370 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal/instance:
uuid: "66670edd7d0b45e9a25dcdf531c5d649"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.526293 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 02:00:49.528635 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:49.529937 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 02:00:49.530094 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal
uuid: "66670edd7d0b45e9a25dcdf531c5d649"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.530197 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:00:49.558624 408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:00:49.559401 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:00:49.559597 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:00:49.567108 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:39725
I20240501 02:00:49.567131 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:39725 every 8 connection(s)
I20240501 02:00:49.568256 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/data/info.pb
I20240501 02:00:49.571271 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:49.572033 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu as pid 408
I20240501 02:00:49.572172 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/master-0/wal/instance
I20240501 02:00:49.577670 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649: Bootstrap starting.
I20240501 02:00:49.580325 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:49.581121 475 log.cc:824] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649: Log is configured to *not* fsync() on all Append() calls
I20240501 02:00:49.582886 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649: No bootstrap required, opened a new log
I20240501 02:00:49.585629 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } }
I20240501 02:00:49.585911 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:49.586020 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66670edd7d0b45e9a25dcdf531c5d649, State: Initialized, Role: FOLLOWER
I20240501 02:00:49.586648 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } }
I20240501 02:00:49.586839 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:00:49.586920 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:00:49.587059 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:49.587993 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } }
I20240501 02:00:49.588353 475 leader_election.cc:302] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66670edd7d0b45e9a25dcdf531c5d649; no voters:
I20240501 02:00:49.588665 475 leader_election.cc:288] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:00:49.589040 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:00:49.589309 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [term 1 LEADER]: Becoming Leader. State: Replica: 66670edd7d0b45e9a25dcdf531c5d649, State: Running, Role: LEADER
I20240501 02:00:49.589702 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } }
I20240501 02:00:49.590077 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:00:49.591306 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66670edd7d0b45e9a25dcdf531c5d649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } } }
I20240501 02:00:49.591413 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [sys.catalog]: This master's current role is: LEADER
I20240501 02:00:49.591728 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66670edd7d0b45e9a25dcdf531c5d649. Latest consensus state: current_term: 1 leader_uuid: "66670edd7d0b45e9a25dcdf531c5d649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66670edd7d0b45e9a25dcdf531c5d649" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39725 } } }
I20240501 02:00:49.591815 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649 [sys.catalog]: This master's current role is: LEADER
I20240501 02:00:49.591910 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:00:49.595207 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:00:49.601047 489 catalog_manager.cc:1347] Generated new cluster ID: cbe108fed91848b6b7c6e3b478ef8f2c
I20240501 02:00:49.601125 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:00:49.611542 489 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:00:49.612542 489 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:00:49.627372 489 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66670edd7d0b45e9a25dcdf531c5d649: Generated new TSK 0
I20240501 02:00:49.628422 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:00:49.638871 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
/tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.1:0
--local_ip_for_outbound_sockets=127.0.99.1
--webserver_interface=127.0.99.1
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39725
--builtin_ntp_servers=127.0.99.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240501 02:00:49.747534 499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:00:49.747784 499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:00:49.747879 499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:00:49.751055 499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:00:49.751161 499 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:00:49.751202 499 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:00:49.751253 499 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:00:49.751334 499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.1
I20240501 02:00:49.755065 499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39725
--never_fsync=true
--heap_profile_path=/tmp/kudu.499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.1
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 01 May 2024 01:43:15 UTC on 1a15035a6830
build id 65
I20240501 02:00:49.755980 499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:00:49.757109 499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:00:49.766017 504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.766469 507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.767323 505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:00:49.768553 499 server_base.cc:1034] running on GCE node
I20240501 02:00:49.768920 499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:00:49.769467 499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:00:49.770612 499 hybrid_clock.cc:648] HybridClock initialized: now 1714528849770600 us; error 27 us; skew 500 ppm
I20240501 02:00:49.772536 499 webserver.cc:460] Webserver started at http://127.0.99.1:36071/ using document root <none> and password file <none>
I20240501 02:00:49.773072 499 fs_manager.cc:359] Metadata directory not provided
I20240501 02:00:49.773128 499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:00:49.773296 499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:00:49.774987 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data/instance:
uuid: "599e15abbc114abe8e917caf739e5ac9"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.775450 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal/instance:
uuid: "599e15abbc114abe8e917caf739e5ac9"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.778746 499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 02:00:49.780854 513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:49.781901 499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 02:00:49.781992 499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal
uuid: "599e15abbc114abe8e917caf739e5ac9"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.782070 499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:00:49.807540 499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:00:49.808192 499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:00:49.808341 499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:00:49.808856 499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:00:49.809743 499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:00:49.809790 499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:49.809865 499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:00:49.809906 499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:49.818660 499 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:37411
I20240501 02:00:49.818681 626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:37411 every 8 connection(s)
I20240501 02:00:49.819628 499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/data/info.pb
I20240501 02:00:49.825165 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu as pid 499
I20240501 02:00:49.825292 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-0/wal/instance
I20240501 02:00:49.827698 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
/tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.2:0
--local_ip_for_outbound_sockets=127.0.99.2
--webserver_interface=127.0.99.2
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39725
--builtin_ntp_servers=127.0.99.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 02:00:49.828616 627 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39725
I20240501 02:00:49.828862 627 heartbeater.cc:464] Registering TS with master...
I20240501 02:00:49.829428 627 heartbeater.cc:511] Master 127.0.99.62:39725 requested a full tablet report, sending...
I20240501 02:00:49.830991 434 ts_manager.cc:194] Registered new tserver with Master: 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411)
I20240501 02:00:49.832765 434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.1:51673
W20240501 02:00:49.936702 630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:00:49.936929 630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:00:49.936992 630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:00:49.940053 630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:00:49.940147 630 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:00:49.940163 630 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:00:49.940183 630 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:00:49.940227 630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.2
I20240501 02:00:49.943858 630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39725
--never_fsync=true
--heap_profile_path=/tmp/kudu.630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.2
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 01 May 2024 01:43:15 UTC on 1a15035a6830
build id 65
I20240501 02:00:49.944762 630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:00:49.945899 630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:00:49.954587 635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.955384 636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:49.956461 638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:00:49.957556 630 server_base.cc:1034] running on GCE node
I20240501 02:00:49.958182 630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:00:49.958915 630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:00:49.960081 630 hybrid_clock.cc:648] HybridClock initialized: now 1714528849960064 us; error 38 us; skew 500 ppm
I20240501 02:00:49.962719 630 webserver.cc:460] Webserver started at http://127.0.99.2:42195/ using document root <none> and password file <none>
I20240501 02:00:49.963315 630 fs_manager.cc:359] Metadata directory not provided
I20240501 02:00:49.963373 630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:00:49.963553 630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:00:49.965235 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data/instance:
uuid: "5b1a962d223b4080b243c1f800804d4b"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.965742 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal/instance:
uuid: "5b1a962d223b4080b243c1f800804d4b"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.969173 630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 02:00:49.971379 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:49.972404 630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 02:00:49.972513 630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal
uuid: "5b1a962d223b4080b243c1f800804d4b"
format_stamp: "Formatted at 2024-05-01 02:00:49 on dist-test-slave-t92c"
I20240501 02:00:49.972595 630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:00:50.002405 630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:00:50.003090 630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:00:50.003262 630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:00:50.003839 630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:00:50.004762 630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:00:50.004812 630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:50.004873 630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:00:50.004904 630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:50.014679 630 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:38843
I20240501 02:00:50.014703 757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:38843 every 8 connection(s)
I20240501 02:00:50.015666 630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/data/info.pb
I20240501 02:00:50.024044 758 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39725
I20240501 02:00:50.024042 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu as pid 630
I20240501 02:00:50.024171 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-1/wal/instance
I20240501 02:00:50.024268 758 heartbeater.cc:464] Registering TS with master...
I20240501 02:00:50.024780 758 heartbeater.cc:511] Master 127.0.99.62:39725 requested a full tablet report, sending...
I20240501 02:00:50.025780 434 ts_manager.cc:194] Registered new tserver with Master: 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843)
I20240501 02:00:50.026278 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
/tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.3:0
--local_ip_for_outbound_sockets=127.0.99.3
--webserver_interface=127.0.99.3
--webserver_port=0
--tserver_master_addrs=127.0.99.62:39725
--builtin_ntp_servers=127.0.99.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 02:00:50.026436 434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.2:52151
W20240501 02:00:50.136371 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:00:50.136605 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:00:50.136679 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:00:50.139654 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:00:50.139737 761 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 02:00:50.139753 761 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 02:00:50.139775 761 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 02:00:50.139823 761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.3
I20240501 02:00:50.143302 761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.99.62:39725
--never_fsync=true
--heap_profile_path=/tmp/kudu.761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.3
--log_dir=/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 01 May 2024 01:43:15 UTC on 1a15035a6830
build id 65
I20240501 02:00:50.144203 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:00:50.145265 761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:00:50.153717 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:50.154762 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:00:50.155715 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:00:50.156487 761 server_base.cc:1034] running on GCE node
I20240501 02:00:50.156816 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:00:50.157326 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:00:50.158463 761 hybrid_clock.cc:648] HybridClock initialized: now 1714528850158452 us; error 25 us; skew 500 ppm
I20240501 02:00:50.160250 761 webserver.cc:460] Webserver started at http://127.0.99.3:37577/ using document root <none> and password file <none>
I20240501 02:00:50.160761 761 fs_manager.cc:359] Metadata directory not provided
I20240501 02:00:50.160811 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:00:50.160964 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:00:50.162642 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data/instance:
uuid: "5bda5b10f348421fa85c2308a797de43"
format_stamp: "Formatted at 2024-05-01 02:00:50 on dist-test-slave-t92c"
I20240501 02:00:50.163108 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal/instance:
uuid: "5bda5b10f348421fa85c2308a797de43"
format_stamp: "Formatted at 2024-05-01 02:00:50 on dist-test-slave-t92c"
I20240501 02:00:50.166299 761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 02:00:50.168363 775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:50.169350 761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 02:00:50.169440 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal
uuid: "5bda5b10f348421fa85c2308a797de43"
format_stamp: "Formatted at 2024-05-01 02:00:50 on dist-test-slave-t92c"
I20240501 02:00:50.169525 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:00:50.178625 761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:00:50.179124 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:00:50.179265 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:00:50.179721 761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:00:50.180509 761 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:00:50.180553 761 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:50.180593 761 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:00:50.180622 761 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:00:50.189616 761 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:45991
I20240501 02:00:50.189641 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:45991 every 8 connection(s)
I20240501 02:00:50.190587 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/data/info.pb
I20240501 02:00:50.192119 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKy4Fl7/build/debug/bin/kudu as pid 761
I20240501 02:00:50.192231 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKy4Fl7/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714528849345733-396-0/linked-list-cluster/ts-2/wal/instance
I20240501 02:00:50.199190 889 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39725
I20240501 02:00:50.199393 889 heartbeater.cc:464] Registering TS with master...
I20240501 02:00:50.199863 889 heartbeater.cc:511] Master 127.0.99.62:39725 requested a full tablet report, sending...
I20240501 02:00:50.200765 434 ts_manager.cc:194] Registered new tserver with Master: 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:50.201265 434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.3:57739
I20240501 02:00:50.206568 396 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:00:50.218503 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58910:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240501 02:00:50.221396 434 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:00:50.238557 822 tablet_service.cc:1467] Processing CreateTablet for tablet 860caa6193cb457d837cdcc4d7ad043e (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:00:50.238557 560 tablet_service.cc:1467] Processing CreateTablet for tablet 860caa6193cb457d837cdcc4d7ad043e (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:00:50.238693 559 tablet_service.cc:1467] Processing CreateTablet for tablet db49c4921d3c4a46b9f47507cc475d6c (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:00:50.239797 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860caa6193cb457d837cdcc4d7ad043e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.239797 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860caa6193cb457d837cdcc4d7ad043e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.240161 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db49c4921d3c4a46b9f47507cc475d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.240389 691 tablet_service.cc:1467] Processing CreateTablet for tablet 860caa6193cb457d837cdcc4d7ad043e (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 02:00:50.241416 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860caa6193cb457d837cdcc4d7ad043e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.242111 690 tablet_service.cc:1467] Processing CreateTablet for tablet db49c4921d3c4a46b9f47507cc475d6c (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:00:50.242367 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db49c4921d3c4a46b9f47507cc475d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.238559 821 tablet_service.cc:1467] Processing CreateTablet for tablet db49c4921d3c4a46b9f47507cc475d6c (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 02:00:50.243247 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db49c4921d3c4a46b9f47507cc475d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.242120 692 tablet_service.cc:1467] Processing CreateTablet for tablet e1813f5ac1904e7c87f93cac584d3e14 (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:00:50.243471 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1813f5ac1904e7c87f93cac584d3e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.243844 561 tablet_service.cc:1467] Processing CreateTablet for tablet e1813f5ac1904e7c87f93cac584d3e14 (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:00:50.244102 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1813f5ac1904e7c87f93cac584d3e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.247298 907 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Bootstrap starting.
I20240501 02:00:50.238559 823 tablet_service.cc:1467] Processing CreateTablet for tablet e1813f5ac1904e7c87f93cac584d3e14 (DEFAULT_TABLE table=TestTable [id=d6fcd1477da54ec1ac7c76d74159ace0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 02:00:50.247649 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1813f5ac1904e7c87f93cac584d3e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:00:50.248760 906 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Bootstrap starting.
I20240501 02:00:50.249549 907 tablet_bootstrap.cc:654] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.250651 907 log.cc:824] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:00:50.250699 906 tablet_bootstrap.cc:654] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.251767 906 log.cc:824] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:00:50.252489 907 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: No bootstrap required, opened a new log
I20240501 02:00:50.252667 907 ts_tablet_manager.cc:1367] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 02:00:50.253302 910 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: Bootstrap starting.
I20240501 02:00:50.254257 906 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: No bootstrap required, opened a new log
I20240501 02:00:50.254382 906 ts_tablet_manager.cc:1367] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 02:00:50.255323 910 tablet_bootstrap.cc:654] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.255945 907 raft_consensus.cc:348] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.256137 907 raft_consensus.cc:374] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.256186 907 raft_consensus.cc:729] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1a962d223b4080b243c1f800804d4b, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.256311 910 log.cc:824] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: Log is configured to *not* fsync() on all Append() calls
I20240501 02:00:50.256915 907 consensus_queue.cc:260] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.256845 906 raft_consensus.cc:348] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.257035 906 raft_consensus.cc:374] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.257114 906 raft_consensus.cc:729] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599e15abbc114abe8e917caf739e5ac9, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.257529 758 heartbeater.cc:503] Master 127.0.99.62:39725 was elected leader, sending a full tablet report...
I20240501 02:00:50.257513 906 consensus_queue.cc:260] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.257727 907 ts_tablet_manager.cc:1398] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240501 02:00:50.257962 910 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: No bootstrap required, opened a new log
I20240501 02:00:50.258118 910 ts_tablet_manager.cc:1367] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 02:00:50.258247 627 heartbeater.cc:503] Master 127.0.99.62:39725 was elected leader, sending a full tablet report...
I20240501 02:00:50.258318 906 ts_tablet_manager.cc:1398] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 02:00:50.258533 906 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: Bootstrap starting.
I20240501 02:00:50.259442 906 tablet_bootstrap.cc:654] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.260519 907 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: Bootstrap starting.
I20240501 02:00:50.261277 906 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: No bootstrap required, opened a new log
I20240501 02:00:50.261356 906 ts_tablet_manager.cc:1367] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 02:00:50.261476 907 tablet_bootstrap.cc:654] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.261898 906 raft_consensus.cc:348] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.261992 906 raft_consensus.cc:374] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.262024 906 raft_consensus.cc:729] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599e15abbc114abe8e917caf739e5ac9, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.262141 906 consensus_queue.cc:260] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.262223 910 raft_consensus.cc:348] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.262372 906 ts_tablet_manager.cc:1398] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:00:50.262395 910 raft_consensus.cc:374] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.262456 910 raft_consensus.cc:729] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bda5b10f348421fa85c2308a797de43, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.262539 906 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: Bootstrap starting.
I20240501 02:00:50.262643 907 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: No bootstrap required, opened a new log
I20240501 02:00:50.262718 907 ts_tablet_manager.cc:1367] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:00:50.263150 910 consensus_queue.cc:260] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.263244 907 raft_consensus.cc:348] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.263338 907 raft_consensus.cc:374] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.263370 907 raft_consensus.cc:729] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1a962d223b4080b243c1f800804d4b, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.263406 906 tablet_bootstrap.cc:654] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.263531 907 consensus_queue.cc:260] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.263794 907 ts_tablet_manager.cc:1398] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:00:50.263952 907 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: Bootstrap starting.
I20240501 02:00:50.264045 910 ts_tablet_manager.cc:1398] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 02:00:50.264381 910 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: Bootstrap starting.
I20240501 02:00:50.264582 906 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: No bootstrap required, opened a new log
I20240501 02:00:50.264648 906 ts_tablet_manager.cc:1367] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:00:50.264801 907 tablet_bootstrap.cc:654] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.265076 906 raft_consensus.cc:348] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.265170 906 raft_consensus.cc:374] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.265201 906 raft_consensus.cc:729] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599e15abbc114abe8e917caf739e5ac9, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.265352 910 tablet_bootstrap.cc:654] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.265323 906 consensus_queue.cc:260] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.265676 906 ts_tablet_manager.cc:1398] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:00:50.265992 889 heartbeater.cc:503] Master 127.0.99.62:39725 was elected leader, sending a full tablet report...
I20240501 02:00:50.266453 910 tablet_bootstrap.cc:492] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: No bootstrap required, opened a new log
I20240501 02:00:50.266520 910 ts_tablet_manager.cc:1367] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20240501 02:00:50.267011 759 tablet.cc:2369] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:00:50.267297 910 raft_consensus.cc:348] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.267477 910 raft_consensus.cc:374] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.267555 910 raft_consensus.cc:729] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bda5b10f348421fa85c2308a797de43, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.267897 910 consensus_queue.cc:260] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.268317 910 ts_tablet_manager.cc:1398] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:00:50.268325 907 tablet_bootstrap.cc:492] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: No bootstrap required, opened a new log
I20240501 02:00:50.268409 907 ts_tablet_manager.cc:1367] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 02:00:50.268469 910 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43: Bootstrap starting.
I20240501 02:00:50.269205 907 raft_consensus.cc:348] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.269318 907 raft_consensus.cc:374] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.269359 907 raft_consensus.cc:729] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1a962d223b4080b243c1f800804d4b, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.269477 910 tablet_bootstrap.cc:654] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43: Neither blocks nor log segments found. Creating new log.
I20240501 02:00:50.269469 907 consensus_queue.cc:260] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.269719 907 ts_tablet_manager.cc:1398] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 02:00:50.270787 910 tablet_bootstrap.cc:492] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43: No bootstrap required, opened a new log
I20240501 02:00:50.270869 910 ts_tablet_manager.cc:1367] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:00:50.271581 910 raft_consensus.cc:348] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.271685 910 raft_consensus.cc:374] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:00:50.271728 910 raft_consensus.cc:729] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bda5b10f348421fa85c2308a797de43, State: Initialized, Role: FOLLOWER
I20240501 02:00:50.271883 910 consensus_queue.cc:260] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.272189 910 ts_tablet_manager.cc:1398] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240501 02:00:50.320847 628 tablet.cc:2369] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:00:50.377082 912 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:00:50.377312 912 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.378441 912 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:50.381971 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1813f5ac1904e7c87f93cac584d3e14" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bda5b10f348421fa85c2308a797de43" is_pre_election: true
I20240501 02:00:50.382273 843 raft_consensus.cc:2454] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 0.
I20240501 02:00:50.382742 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1813f5ac1904e7c87f93cac584d3e14" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9" is_pre_election: true
I20240501 02:00:50.382750 645 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.382984 581 raft_consensus.cc:2454] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 0.
I20240501 02:00:50.382982 912 raft_consensus.cc:2781] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:00:50.383059 912 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:00:50.383098 912 raft_consensus.cc:3037] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.384099 912 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.384425 912 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 election: Requested vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:50.384745 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1813f5ac1904e7c87f93cac584d3e14" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9"
I20240501 02:00:50.384848 581 raft_consensus.cc:3037] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.384826 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1813f5ac1904e7c87f93cac584d3e14" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bda5b10f348421fa85c2308a797de43"
I20240501 02:00:50.384972 843 raft_consensus.cc:3037] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.385911 843 raft_consensus.cc:2454] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 1.
I20240501 02:00:50.386154 581 raft_consensus.cc:2454] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 1.
I20240501 02:00:50.386198 645 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.386361 912 raft_consensus.cc:2781] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:00:50.386593 912 raft_consensus.cc:686] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [term 1 LEADER]: Becoming Leader. State: Replica: 5b1a962d223b4080b243c1f800804d4b, State: Running, Role: LEADER
I20240501 02:00:50.386899 912 consensus_queue.cc:237] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.389760 439 catalog_manager.cc:5484] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1a962d223b4080b243c1f800804d4b (127.0.99.2). New cstate: current_term: 1 leader_uuid: "5b1a962d223b4080b243c1f800804d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } health_report { overall_health: UNKNOWN } } }
I20240501 02:00:50.409840 912 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:00:50.410012 912 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.410455 912 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:50.410933 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db49c4921d3c4a46b9f47507cc475d6c" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bda5b10f348421fa85c2308a797de43" is_pre_election: true
I20240501 02:00:50.410925 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db49c4921d3c4a46b9f47507cc475d6c" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9" is_pre_election: true
I20240501 02:00:50.411064 843 raft_consensus.cc:2454] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 0.
I20240501 02:00:50.411079 581 raft_consensus.cc:2454] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 0.
I20240501 02:00:50.411360 645 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.411546 912 raft_consensus.cc:2781] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:00:50.411607 912 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:00:50.411653 912 raft_consensus.cc:3037] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.412452 912 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.412725 912 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 election: Requested vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:50.413111 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db49c4921d3c4a46b9f47507cc475d6c" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9"
I20240501 02:00:50.413110 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db49c4921d3c4a46b9f47507cc475d6c" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bda5b10f348421fa85c2308a797de43"
I20240501 02:00:50.413228 581 raft_consensus.cc:3037] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.413228 843 raft_consensus.cc:3037] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.414155 581 raft_consensus.cc:2454] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 1.
I20240501 02:00:50.414155 843 raft_consensus.cc:2454] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1a962d223b4080b243c1f800804d4b in term 1.
I20240501 02:00:50.414444 645 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.414630 912 raft_consensus.cc:2781] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:00:50.414709 912 raft_consensus.cc:686] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [term 1 LEADER]: Becoming Leader. State: Replica: 5b1a962d223b4080b243c1f800804d4b, State: Running, Role: LEADER
I20240501 02:00:50.414861 912 consensus_queue.cc:237] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.416582 439 catalog_manager.cc:5484] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1a962d223b4080b243c1f800804d4b (127.0.99.2). New cstate: current_term: 1 leader_uuid: "5b1a962d223b4080b243c1f800804d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } health_report { overall_health: UNKNOWN } } }
W20240501 02:00:50.441529 890 tablet.cc:2369] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:00:50.547935 916 raft_consensus.cc:482] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:00:50.548084 916 raft_consensus.cc:504] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.549149 916 leader_election.cc:288] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843)
I20240501 02:00:50.552894 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "5bda5b10f348421fa85c2308a797de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9" is_pre_election: true
I20240501 02:00:50.553081 581 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bda5b10f348421fa85c2308a797de43 in term 0.
I20240501 02:00:50.553351 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "5bda5b10f348421fa85c2308a797de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1a962d223b4080b243c1f800804d4b" is_pre_election: true
I20240501 02:00:50.553438 779 leader_election.cc:302] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.553570 712 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bda5b10f348421fa85c2308a797de43 in term 0.
I20240501 02:00:50.553701 916 raft_consensus.cc:2781] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:00:50.553781 916 raft_consensus.cc:482] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:00:50.553853 916 raft_consensus.cc:3037] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.554711 916 raft_consensus.cc:504] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.555034 916 leader_election.cc:288] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [CANDIDATE]: Term 1 election: Requested vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843)
I20240501 02:00:50.555368 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "5bda5b10f348421fa85c2308a797de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1a962d223b4080b243c1f800804d4b"
I20240501 02:00:50.555368 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "5bda5b10f348421fa85c2308a797de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9"
I20240501 02:00:50.555481 712 raft_consensus.cc:3037] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.555481 581 raft_consensus.cc:3037] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:00:50.556401 712 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bda5b10f348421fa85c2308a797de43 in term 1.
I20240501 02:00:50.556402 581 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bda5b10f348421fa85c2308a797de43 in term 1.
I20240501 02:00:50.556665 779 leader_election.cc:302] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43; no voters:
I20240501 02:00:50.556830 916 raft_consensus.cc:2781] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:00:50.557062 916 raft_consensus.cc:686] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [term 1 LEADER]: Becoming Leader. State: Replica: 5bda5b10f348421fa85c2308a797de43, State: Running, Role: LEADER
I20240501 02:00:50.557397 916 consensus_queue.cc:237] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:50.560245 439 catalog_manager.cc:5484] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bda5b10f348421fa85c2308a797de43 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "5bda5b10f348421fa85c2308a797de43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } health_report { overall_health: HEALTHY } } }
I20240501 02:00:50.602742 396 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:00:50.604416 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 599e15abbc114abe8e917caf739e5ac9 to finish bootstrapping
I20240501 02:00:50.610535 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5b1a962d223b4080b243c1f800804d4b to finish bootstrapping
I20240501 02:00:50.615160 396 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5bda5b10f348421fa85c2308a797de43 to finish bootstrapping
I20240501 02:00:50.620476 930 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.0.99.62:42445/mem-trackers
http://127.0.99.1:36071/mem-trackers
http://127.0.99.1:36071/scans
http://127.0.99.62:42445/metrics
http://127.0.99.1:36071/threadz
http://127.0.99.2:42195/tablets
http://127.0.99.62:42445/dump-entities
http://127.0.99.2:42195/threadz?group=all
http://127.0.99.3:37577/threadz?group=all
http://127.0.99.62:42445/tablet-servers
http://127.0.99.62:42445/threadz?group=all
http://127.0.99.3:37577/maintenance-manager
http://127.0.99.1:36071/transactions?tablet_id=db49c4921d3c4a46b9f47507cc475d6c
http://127.0.99.2:42195/threadz
http://127.0.99.2:42195/transactions?tablet_id=db49c4921d3c4a46b9f47507cc475d6c
http://127.0.99.62:42445/tables
http://127.0.99.3:37577/startup
http://127.0.99.3:37577/tablets
http://127.0.99.3:37577/stacks
http://127.0.99.3:37577/mem-trackers
http://127.0.99.3:37577/scans
http://127.0.99.62:42445/startup
http://127.0.99.2:42195/maintenance-manager
http://127.0.99.3:37577/transactions?tablet_id=db49c4921d3c4a46b9f47507cc475d6c
http://127.0.99.1:36071/metrics
http://127.0.99.1:36071/stacks
http://127.0.99.62:42445/stacks
http://127.0.99.62:42445/masters
http://127.0.99.1:36071/threadz?group=all
http://127.0.99.1:36071/startup
http://127.0.99.1:36071/maintenance-manager
http://127.0.99.2:42195/stacks
http://127.0.99.1:36071/tablets
http://127.0.99.2:42195/scans
http://127.0.99.62:42445/ipki-ca-cert
http://127.0.99.3:37577/threadz
http://127.0.99.62:42445/ipki-ca-cert-der
http://127.0.99.3:37577/metrics
http://127.0.99.2:42195/metrics
http://127.0.99.62:42445/ipki-ca-cert-pem
http://127.0.99.62:42445/threadz
http://127.0.99.2:42195/startup
http://127.0.99.2:42195/mem-trackers
I20240501 02:00:50.623786 396 linked_list-test-util.h:401] Running for: 30.000s
I20240501 02:00:50.623855 396 linked_list-test-util.h:402] Sampling every 10000000 us
I20240501 02:00:50.623872 396 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240501 02:00:50.626745 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:50.640506 712 raft_consensus.cc:1261] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Refusing update from remote peer 5bda5b10f348421fa85c2308a797de43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.640609 843 raft_consensus.cc:1261] T db49c4921d3c4a46b9f47507cc475d6c P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Refusing update from remote peer 5b1a962d223b4080b243c1f800804d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.640614 581 raft_consensus.cc:1261] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Refusing update from remote peer 5b1a962d223b4080b243c1f800804d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.640995 842 raft_consensus.cc:1261] T e1813f5ac1904e7c87f93cac584d3e14 P 5bda5b10f348421fa85c2308a797de43 [term 1 FOLLOWER]: Refusing update from remote peer 5b1a962d223b4080b243c1f800804d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.641054 580 raft_consensus.cc:1261] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Refusing update from remote peer 5b1a962d223b4080b243c1f800804d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.641077 916 consensus_queue.cc:1035] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.641403 923 consensus_queue.cc:1035] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.641479 939 consensus_queue.cc:1035] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.641760 923 consensus_queue.cc:1035] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.641791 912 consensus_queue.cc:1035] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.642570 579 raft_consensus.cc:1261] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Refusing update from remote peer 5bda5b10f348421fa85c2308a797de43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:00:50.644040 916 consensus_queue.cc:1035] T 860caa6193cb457d837cdcc4d7ad043e P 5bda5b10f348421fa85c2308a797de43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:00:50.648900 938 mvcc.cc:205] Tried to move back new op lower bound from 7022710172216893440 to 7022710171300712448. Current Snapshot: MvccSnapshot[applied={T|T < 7022710172216893440}]
I20240501 02:00:50.652396 941 mvcc.cc:205] Tried to move back new op lower bound from 7022710172216983552 to 7022710171885834240. Current Snapshot: MvccSnapshot[applied={T|T < 7022710172216983552}]
I20240501 02:00:50.649528 937 mvcc.cc:205] Tried to move back new op lower bound from 7022710172216893440 to 7022710171300712448. Current Snapshot: MvccSnapshot[applied={T|T < 7022710172216893440}]
I20240501 02:00:50.737401 931 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/902/stat: No such file or directory (error 2)
I20240501 02:00:51.350945 963 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 02:00:51.356017 963 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fa593100dcd google::LogMessage::Fail() at ??:0
@ 0x7fa593104b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa5931007cc google::LogMessage::Flush() at ??:0
@ 0x7fa593101f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa5944e25b1 kudu::operator-() at ??:0
@ 0x7fa5965c5fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fa5965af05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fa5965aef90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fa5965859bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fa5971b185d kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fa5971b0936 _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fa5971b9145 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fa59207bd09 std::function<>::operator()() at ??:0
@ 0x7fa592076107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fa592075a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fa592075123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fa59209db89 handle_request at ??:0
@ 0x7fa5920a026b worker_thread at ??:0
@ 0x7fa594d7f6db start_thread at ??:0
@ 0x7fa5922b771f clone at ??:0
W20240501 02:00:51.565202 892 meta_cache.cc:302] tablet 860caa6193cb457d837cdcc4d7ad043e: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: recv got EOF from 127.0.99.3:45991 (error 108)
W20240501 02:00:51.574858 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.580637 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.597923 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.606597 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.625339 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.639313 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.681200 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.685075 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.688161 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.692804 645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:00:51.694954 645 consensus_peers.cc:485] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:00:51.695822 645 consensus_peers.cc:485] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:00:51.710244 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.711935 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.721262 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.737783 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.744943 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.746263 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.763556 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.771209 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.786068 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.808229 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.821916 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.827451 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.840245 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.869589 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.880735 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.894338 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.901074 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.946413 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.947098 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:51.969606 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:51.976236 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.017387 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.031723 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.042409 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.062361 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.097247 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.125103 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.129979 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.163477 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.185279 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.215935 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.232581 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.270231 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.285502 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.318941 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.343791 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.381069 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.391673 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.427269 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.468309 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.505939 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.508327 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.549073 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.592367 892 meta_cache.cc:302] tablet 860caa6193cb457d837cdcc4d7ad043e: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:00:52.593438 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.636234 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.637298 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.677937 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.732340 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.768291 540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:00:52.777901 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
W20240501 02:00:52.814953 672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57168: Illegal state: replica 5b1a962d223b4080b243c1f800804d4b is not leader of this config: current role FOLLOWER
I20240501 02:00:52.849568 912 raft_consensus.cc:482] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5bda5b10f348421fa85c2308a797de43)
I20240501 02:00:52.849730 912 raft_consensus.cc:504] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:52.850262 912 leader_election.cc:288] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1:37411), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:52.851125 578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "5b1a962d223b4080b243c1f800804d4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 55 } } ignore_live_leader: false dest_uuid: "599e15abbc114abe8e917caf739e5ac9" is_pre_election: true
W20240501 02:00:52.851569 645 leader_election.cc:334] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:00:52.851790 992 raft_consensus.cc:482] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5bda5b10f348421fa85c2308a797de43)
I20240501 02:00:52.851881 992 raft_consensus.cc:504] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:52.852262 648 leader_election.cc:302] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b1a962d223b4080b243c1f800804d4b; no voters: 599e15abbc114abe8e917caf739e5ac9, 5bda5b10f348421fa85c2308a797de43
I20240501 02:00:52.852475 912 raft_consensus.cc:2726] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:00:52.853021 992 leader_election.cc:288] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:00:52.853611 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:00:52.854442 514 leader_election.cc:334] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:00:52.857791 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "599e15abbc114abe8e917caf739e5ac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 55 } } ignore_live_leader: false dest_uuid: "5b1a962d223b4080b243c1f800804d4b" is_pre_election: true
I20240501 02:00:52.857939 711 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599e15abbc114abe8e917caf739e5ac9 in term 1.
I20240501 02:00:52.858291 517 leader_election.cc:302] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9, 5b1a962d223b4080b243c1f800804d4b; no voters: 5bda5b10f348421fa85c2308a797de43
I20240501 02:00:52.858542 992 raft_consensus.cc:2781] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 02:00:52.858599 992 raft_consensus.cc:482] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5bda5b10f348421fa85c2308a797de43)
I20240501 02:00:52.858650 992 raft_consensus.cc:3037] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:00:52.859436 992 raft_consensus.cc:504] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:00:52.859705 992 leader_election.cc:288] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 election: Requested vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
I20240501 02:00:52.860162 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860caa6193cb457d837cdcc4d7ad043e" candidate_uuid: "599e15abbc114abe8e917caf739e5ac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 55 } } ignore_live_leader: false dest_uuid: "5b1a962d223b4080b243c1f800804d4b"
I20240501 02:00:52.860280 711 raft_consensus.cc:3037] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:00:52.860994 711 raft_consensus.cc:2454] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599e15abbc114abe8e917caf739e5ac9 in term 2.
I20240501 02:00:52.861285 517 leader_election.cc:302] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9, 5b1a962d223b4080b243c1f800804d4b; no voters:
I20240501 02:00:52.861425 992 raft_consensus.cc:2781] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:00:52.861599 992 raft_consensus.cc:686] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [term 2 LEADER]: Becoming Leader. State: Replica: 599e15abbc114abe8e917caf739e5ac9, State: Running, Role: LEADER
I20240501 02:00:52.861928 992 consensus_queue.cc:237] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
W20240501 02:00:52.863091 514 leader_election.cc:334] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:00:52.864598 436 catalog_manager.cc:5484] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 reported cstate change: term changed from 1 to 2, leader changed from 5bda5b10f348421fa85c2308a797de43 (127.0.99.3) to 599e15abbc114abe8e917caf739e5ac9 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "599e15abbc114abe8e917caf739e5ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } health_report { overall_health: UNKNOWN } } }
I20240501 02:00:52.878712 711 raft_consensus.cc:1261] T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b [term 2 FOLLOWER]: Refusing update from remote peer 599e15abbc114abe8e917caf739e5ac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 2 index: 57. (index mismatch)
I20240501 02:00:52.879230 997 consensus_queue.cc:1035] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
W20240501 02:00:52.879489 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 02:00:52.902791 952 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:52.937331 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:00:53.946290 645 consensus_peers.cc:485] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 02:00:54.155702 645 consensus_peers.cc:485] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 02:00:54.839111 952 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:54.871237 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:55.024479 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.000000
I20240501 02:00:55.082048 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.000000
W20240501 02:00:55.181279 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 02:00:55.493224 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.468s user 0.249s sys 0.008s Metrics: {"bytes_written":148491,"cfile_init":1,"compiler_manager_pool.queue_time_us":22420,"compiler_manager_pool.run_cpu_time_us":254824,"compiler_manager_pool.run_wall_time_us":324401,"dirs.queue_time_us":9633,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3144,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8164,"thread_start_us":212,"threads_started":2}
I20240501 02:00:55.494546 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 142945 bytes on disk
I20240501 02:00:55.495174 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 02:00:55.761584 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.679s user 0.293s sys 0.004s Metrics: {"bytes_written":149495,"cfile_init":1,"compiler_manager_pool.queue_time_us":65374,"compiler_manager_pool.run_cpu_time_us":265092,"compiler_manager_pool.run_wall_time_us":552204,"dirs.queue_time_us":34449,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15915,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8223,"thread_start_us":188,"threads_started":2}
I20240501 02:00:55.766053 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 143699 bytes on disk
I20240501 02:00:55.766810 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240501 02:00:56.375219 645 consensus_peers.cc:485] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:00:56.619609 645 consensus_peers.cc:485] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:00:56.875381 645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240501 02:00:56.886626 952 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:56.920830 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:00:57.744129 1015 log.cc:924] Time spent T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20240501 02:00:57.744428 1010 log.cc:924] Time spent T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20240501 02:00:57.744450 1007 log.cc:924] Time spent T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20240501 02:00:57.744606 998 log.cc:924] Time spent T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20240501 02:00:57.744736 1005 log.cc:924] Time spent T 860caa6193cb457d837cdcc4d7ad043e P 5b1a962d223b4080b243c1f800804d4b: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20240501 02:00:57.767127 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:00:57.771999 1014 log.cc:924] Time spent T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20240501 02:00:58.210958 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 02:00:58.751586 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.000000
I20240501 02:00:58.774407 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.000000
W20240501 02:00:58.785961 645 consensus_peers.cc:485] T e1813f5ac1904e7c87f93cac584d3e14 P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 02:00:58.955324 952 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:59.003221 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:00:59.083508 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.332s user 0.242s sys 0.004s Metrics: {"bytes_written":156068,"cfile_init":1,"dirs.queue_time_us":12342,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":5624,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1593,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8546,"thread_start_us":93,"threads_started":1}
I20240501 02:00:59.084805 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 135852 bytes on disk
I20240501 02:00:59.085383 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240501 02:00:59.086038 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling CompactRowSetsOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.014816
I20240501 02:00:59.094766 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: FlushMRSOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.320s user 0.230s sys 0.000s Metrics: {"bytes_written":155685,"cfile_init":1,"dirs.queue_time_us":4904,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2358,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1736,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8519,"thread_start_us":76,"threads_started":1}
I20240501 02:00:59.096037 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 135304 bytes on disk
I20240501 02:00:59.097314 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 02:00:59.098414 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling CompactRowSetsOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=1.014816
W20240501 02:00:59.548837 645 consensus_peers.cc:485] T db49c4921d3c4a46b9f47507cc475d6c P 5b1a962d223b4080b243c1f800804d4b -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 02:01:00.103101 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: CompactRowSetsOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 1.014s user 0.426s sys 0.000s Metrics: {"bytes_written":12282,"cfile_cache_hit":13,"cfile_cache_hit_bytes":146619,"cfile_cache_miss":57,"cfile_cache_miss_bytes":935897,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":962,"drs_written":1,"lbm_read_time_us":2316,"lbm_reads_lt_1ms":85,"lbm_write_time_us":2936,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":513137,"reinsert_count":0,"rows_written":16710,"thread_start_us":358,"threads_started":7,"update_count":1898}
I20240501 02:01:00.103988 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling MajorDeltaCompactionOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=0.046771
I20240501 02:01:00.118551 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: CompactRowSetsOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 1.019s user 0.426s sys 0.004s Metrics: {"bytes_written":12478,"cfile_cache_hit":13,"cfile_cache_hit_bytes":147099,"cfile_cache_miss":57,"cfile_cache_miss_bytes":937071,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":638,"drs_written":1,"lbm_read_time_us":2029,"lbm_reads_lt_1ms":85,"lbm_write_time_us":2922,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":512587,"reinsert_count":0,"rows_written":16742,"thread_start_us":389,"threads_started":7,"update_count":1930}
I20240501 02:01:00.119427 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 289716 bytes on disk
I20240501 02:01:00.119910 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 02:01:00.120443 628 maintenance_manager.cc:419] P 599e15abbc114abe8e917caf739e5ac9: Scheduling MajorDeltaCompactionOp(e1813f5ac1904e7c87f93cac584d3e14): perf score=0.047430
I20240501 02:01:00.168285 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: MajorDeltaCompactionOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.064s user 0.051s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14766,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":92,"lbm_read_time_us":368,"lbm_reads_lt_1ms":25,"lbm_write_time_us":276,"lbm_writes_lt_1ms":9,"reinsert_count":0,"update_count":1898}
I20240501 02:01:00.168963 759 maintenance_manager.cc:419] P 5b1a962d223b4080b243c1f800804d4b: Scheduling UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14): 289436 bytes on disk
I20240501 02:01:00.169390 649 maintenance_manager.cc:643] P 5b1a962d223b4080b243c1f800804d4b: UndoDeltaBlockGCOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 02:01:00.185983 518 maintenance_manager.cc:643] P 599e15abbc114abe8e917caf739e5ac9: MajorDeltaCompactionOp(e1813f5ac1904e7c87f93cac584d3e14) complete. Timing: real 0.065s user 0.048s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14964,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":62,"lbm_read_time_us":290,"lbm_reads_lt_1ms":21,"lbm_write_time_us":311,"lbm_writes_lt_1ms":9,"reinsert_count":0,"update_count":1930}
W20240501 02:01:00.203346 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 02:01:00.629943 396 linked_list-test-util.h:417] Sample at HT timestamp: 7022710213115355136 Inserted count: 29100
I20240501 02:01:01.156307 952 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:01:01.189484 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 02:01:01.239949 952 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f48fa01bdcd google::LogMessage::Fail() at ??:0
@ 0x7f48fa01fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f48fa01b7cc google::LogMessage::Flush() at ??:0
@ 0x7f48fa01cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f48fb3fd5b1 kudu::operator-() at ??:0
@ 0x7f48fd4e0fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f48fd4ca05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f48fd4c9f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f48fd4a09bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f48fe0cc85d kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f48fe0cb936 _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7f48fe0d4145 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7f48f8f96d09 std::function<>::operator()() at ??:0
@ 0x7f48f8f91107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f48f8f90a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f48f8f90123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f48f8fb8b89 handle_request at ??:0
@ 0x7f48f8fbb26b worker_thread at ??:0
@ 0x7f48fbc9a6db start_thread at ??:0
@ 0x7f48f91d271f clone at ??:0
W20240501 02:01:01.453783 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
W20240501 02:01:01.458227 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.476006 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.509258 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.549885 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.597913 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.652662 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.664078 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 02:01:01.720016 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.791126 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.880239 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:01.968195 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.072456 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.182410 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.193768 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.199294 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.214126 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.234061 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.255702 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.275970 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.303602 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.320508 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.321781 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:02.354396 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:02.387068 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.391484 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.455623 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.465438 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.476131 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:02.503810 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.554376 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.613349 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.701416 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:02.732197 1102 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:02.732388 1102 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:02.732888 1102 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:02.734167 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:02.734288 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 02:01:02.734454 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:02.734529 517 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:02.734707 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:01:02.744171 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:02.744369 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:02.745271 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:02.746166 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:02.746358 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:02.746425 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:02.746568 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:02.757280 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.773402 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.801064 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.877641 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:02.955516 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.044798 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.076977 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.160770 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.215832 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 02:01:03.249600 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.346715 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:03.354811 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:03.387576 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.491938 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.525907 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:03.604956 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.651927 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.762220 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.874377 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:03.924535 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 02:01:03.932714 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.053791 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.159178 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.244812 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:04.355233 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:04.381817 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.445251 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.533775 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:04.573266 992 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:04.573418 992 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:04.573899 992 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:04.575155 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:04.575260 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:04.575352 514 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:04.575513 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:04.584556 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.691000 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:04.796877 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:04.797431 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:04.797549 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:04.798142 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:04.799173 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:04.799270 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:04.799363 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:04.799539 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:04.940873 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.004017 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 02:01:05.014418 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.165585 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.280182 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:05.355427 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:05.404354 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.551576 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.569924 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:05.653519 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.830158 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:05.910419 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.088568 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.217404 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.355342 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:06.355791 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:06.517268 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.590844 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:06.629128 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.648389 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 02:01:06.819535 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:06.827421 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:07.037635 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:07.207331 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:07.355731 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:07.359409 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:07.474572 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 02:01:07.513538 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:07.617431 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:07.691504 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:07.820180 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:07.820331 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:07.820952 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:07.822147 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:07.827291 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:07.827373 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:07.827657 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:07.828496 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:07.925273 992 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:07.925479 992 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:07.926205 992 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:07.927496 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:07.927599 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:07.927665 517 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:07.927827 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:08.031764 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:08.148121 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:08.355950 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:08.367952 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:08.375592 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:08.474454 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 02:01:08.614953 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:08.677668 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:08.820677 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:08.977842 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:09.125911 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 02:01:09.171054 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:09.350880 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:09.355784 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:01:09.491199 1106 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:09.491359 1106 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:09.491922 1106 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:09.493300 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:09.493515 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:09.493629 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:09.493799 1106 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:09.528465 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:09.592417 992 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:09.592597 992 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:09.593067 992 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:09.594208 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:09.594425 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:09.594535 517 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:09.594709 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:09.734006 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:09.752956 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:09.892378 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:10.076604 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 02:01:10.126421 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:10.266341 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:10.356068 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:10.524550 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:10.650712 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:10.770916 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:10.910611 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:11.077422 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:11.311149 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:11.356187 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:11.375820 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 02:01:11.496198 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:11.716818 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:11.784958 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:11.928705 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:12.136001 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:12.182123 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:12.182283 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:12.182823 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:12.184000 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:12.184175 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:12.184237 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:12.184398 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:01:12.356384 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:12.361582 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:12.566299 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:12.661976 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 02:01:12.807044 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:12.851045 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:13.006371 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:13.264982 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:13.357021 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:13.454285 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:13.715360 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 26 similar messages]
W20240501 02:01:13.725895 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:13.885342 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:13.905318 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:14.019275 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 02:01:14.203456 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:14.356931 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:14.369446 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:14.686132 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:14.690944 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:14.940656 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:14.940802 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:14.941287 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:14.942422 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:14.942539 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:14.942644 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:14.942837 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:14.987069 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:15.018014 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:15.269438 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 02:01:15.324103 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:15.356642 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:15.514240 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:15.815963 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:16.021672 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:16.125654 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:16.196547 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 02:01:16.312875 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:16.357072 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:16.541980 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:16.807509 992 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:16.807680 992 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:16.808355 992 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:16.809535 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:16.809684 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:16.809746 514 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:16.809903 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:16.822001 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:17.066437 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:17.144320 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:17.340667 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:17.357059 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:17.598944 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:17.716019 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 02:01:17.867169 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:18.140650 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:18.199409 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:18.357230 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:18.405050 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:18.422350 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 02:01:18.691272 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:18.885907 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240501 02:01:18.945981 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:19.251367 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:19.291070 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:19.357452 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:19.494096 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:19.674073 992 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:19.674217 992 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:19.674961 992 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:19.676198 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:19.676293 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:19.676384 517 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:19.676549 992 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:19.826902 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:20.057116 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:20.188608 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 02:01:20.357380 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:20.410514 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:20.424270 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:20.629408 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:20.885672 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 02:01:20.999737 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:21.204830 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:21.357693 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:21.591866 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:21.595724 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:21.791401 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:22.187954 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:22.358158 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:22.404317 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:22.570775 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 02:01:22.609541 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:22.792506 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:23.024403 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:23.358062 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:23.407601 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:23.461066 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 02:01:23.610013 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:23.652328 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:24.029670 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:24.073727 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 02:01:24.289477 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:24.358284 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 02:01:24.412192 992 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:24.412362 992 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:24.412837 992 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:24.414211 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:24.414593 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:24.414690 517 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:24.414872 992 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:24.659021 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:24.660269 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:24.938020 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:25.261200 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 02:01:25.296401 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:25.358443 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:25.594493 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:25.727363 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:25.946190 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:26.109838 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 02:01:26.258984 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:26.358616 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:26.606422 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:26.822733 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:26.931715 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:27.270925 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:27.358784 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:27.612087 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:27.816167 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 02:01:27.841348 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:27.949565 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:28.306561 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:28.358840 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:28.509290 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 02:01:28.639151 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:28.756428 1115 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:28.756584 1115 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:28.757063 1115 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:28.758396 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:28.758621 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:28.758719 517 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:28.758914 1115 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:28.864310 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:29.006927 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:29.287482 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240501 02:01:29.327409 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:29.358858 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:29.711439 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:29.948133 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:30.030943 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:30.209460 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 02:01:30.359165 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:30.428951 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:30.744930 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:30.984481 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:31.105091 1115 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:31.105247 1115 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:31.105738 1115 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:31.107214 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:31.107368 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:31.107430 517 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:31.107583 1115 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:31.160874 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:31.232681 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 02:01:31.359253 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:31.463604 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:31.897779 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:32.143941 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:32.193496 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:32.359356 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:32.643909 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:32.725486 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 02:01:32.931259 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:33.145025 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:33.359710 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:33.397353 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:33.654999 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 02:01:33.685743 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:34.158329 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:34.191423 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:34.343768 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 02:01:34.359709 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:34.443351 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:34.924978 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:35.030045 1121 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:35.030273 1121 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:35.030995 1121 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:35.032361 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:35.032716 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:35.032815 514 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:35.033036 1121 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:35.209091 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:35.219625 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:35.359680 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:35.373123 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 02:01:35.707665 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:35.979744 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:36.029630 1121 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:36.029784 1121 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:36.030303 1121 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:36.031694 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:36.031776 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:36.031831 514 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:36.032017 1121 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:36.177170 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 02:01:36.231026 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:36.360138 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:36.495328 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:37.022013 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:37.286299 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:37.295945 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:37.360193 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:37.654645 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 02:01:37.818517 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:38.104270 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:38.355650 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:38.359954 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:38.624994 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:38.802002 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 02:01:38.920192 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:39.344597 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240501 02:01:39.360350 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:39.438187 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:39.461055 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:39.741027 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:40.213372 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 02:01:40.256359 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:40.360502 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:40.532210 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:40.577059 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:41.087888 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:41.286038 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 02:01:41.360662 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:41.416509 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:41.648295 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:41.927196 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:42.271155 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:42.360801 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:42.779157 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:42.809736 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:42.878767 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 02:01:43.137171 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:43.360711 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:43.637765 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:43.897631 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 02:01:43.923866 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:44.005626 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:44.360961 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:44.422300 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 02:01:44.499891 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:44.888726 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:45.083698 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:45.360821 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 02:01:45.361155 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:45.374176 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:45.774428 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:46.259116 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:46.305676 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:46.361294 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:46.616351 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 02:01:46.672524 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:47.111325 1127 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:47.111528 1127 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:47.112136 1127 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:47.113415 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:47.113571 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:47.113627 514 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:47.113791 1127 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:47.153371 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:47.361074 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:47.451735 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:47.576598 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:47.866353 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 02:01:48.055982 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:48.361450 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:48.487450 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:48.662461 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:48.966931 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:49.310134 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 02:01:49.361578 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:49.416185 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:49.727695 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:49.825166 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 02:01:49.892815 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:50.353380 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:50.361577 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:50.412712 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 02:01:50.821257 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:50.872100 892 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:51.299358 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:51.361557 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:51.761179 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:51.930670 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:51.991966 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 02:01:52.248345 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:52.362114 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:52.709697 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:53.029291 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:53.139191 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 02:01:53.207125 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:53.362000 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:53.666237 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:54.041051 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:54.178145 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:54.362314 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:54.412283 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 02:01:54.635804 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:54.974017 517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 02:01:55.127799 1132 raft_consensus.cc:482] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:55.127946 1132 raft_consensus.cc:504] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:55.128446 1132 leader_election.cc:288] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:55.129693 514 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:55.129777 517 leader_election.cc:334] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:55.129864 517 leader_election.cc:302] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:55.130044 1132 raft_consensus.cc:2726] T db49c4921d3c4a46b9f47507cc475d6c P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:55.159128 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:55.288201 895 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:55.362627 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:55.613924 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:55.742782 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 02:01:56.146956 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:56.317781 895 meta_cache.cc:302] tablet db49c4921d3c4a46b9f47507cc475d6c: replica 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
I20240501 02:01:56.362581 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:56.595655 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:56.884464 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 02:01:56.968108 1132 raft_consensus.cc:482] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b1a962d223b4080b243c1f800804d4b)
I20240501 02:01:56.968261 1132 raft_consensus.cc:504] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599e15abbc114abe8e917caf739e5ac9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37411 } } peers { permanent_uuid: "5b1a962d223b4080b243c1f800804d4b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38843 } } peers { permanent_uuid: "5bda5b10f348421fa85c2308a797de43" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45991 } }
I20240501 02:01:56.968751 1132 leader_election.cc:288] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843), 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991)
W20240501 02:01:56.969991 517 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111)
W20240501 02:01:56.970086 514 leader_election.cc:334] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
I20240501 02:01:56.970201 514 leader_election.cc:302] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599e15abbc114abe8e917caf739e5ac9; no voters: 5b1a962d223b4080b243c1f800804d4b, 5bda5b10f348421fa85c2308a797de43
I20240501 02:01:56.970381 1132 raft_consensus.cc:2726] T e1813f5ac1904e7c87f93cac584d3e14 P 599e15abbc114abe8e917caf739e5ac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 02:01:57.144825 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:57.362702 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:57.478406 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:57.587293 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:58.153920 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:58.249562 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 02:01:58.362921 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:58.490808 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:58.591818 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:01:59.165409 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:01:59.363124 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:01:59.425572 517 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5b1a962d223b4080b243c1f800804d4b (127.0.99.2:38843): Couldn't send request to peer 5b1a962d223b4080b243c1f800804d4b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.2:38843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 02:01:59.506819 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:01:59.606772 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:02:00.190649 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:02:00.363003 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:02:00.476477 514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240501 02:02:00.532181 892 meta_cache.cc:302] tablet e1813f5ac1904e7c87f93cac584d3e14: replica 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111)
W20240501 02:02:00.630043 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
W20240501 02:02:01.089880 514 consensus_peers.cc:485] T 860caa6193cb457d837cdcc4d7ad043e P 599e15abbc114abe8e917caf739e5ac9 -> Peer 5bda5b10f348421fa85c2308a797de43 (127.0.99.3:45991): Couldn't send request to peer 5bda5b10f348421fa85c2308a797de43. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.3:45991: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 02:02:01.222944 541 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50392: Illegal state: replica 599e15abbc114abe8e917caf739e5ac9 is not leader of this config: current role FOLLOWER
I20240501 02:02:01.363349 932 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 02:02:01.568782 895 batcher.cc:441] Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569357 396 client-test-util.cc:49] 35 failed ops. First 10 errors follow
I20240501 02:02:01.569456 396 client-test-util.cc:55] Op INSERT int64 rand_key=-3636930612703199232, int64 link_to=8538346272613400576, int64 insert_ts=1714528861236390 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569653 396 client-test-util.cc:55] Op INSERT int64 rand_key=-3685571046346588159, int64 link_to=3695877934237089793, int64 insert_ts=1714528861236517 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569695 396 client-test-util.cc:55] Op INSERT int64 rand_key=-3734492954966687742, int64 link_to=-1146871879115931646, int64 insert_ts=1714528861236573 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569761 396 client-test-util.cc:55] Op INSERT int64 rand_key=-3783274126098366461, int64 link_to=-5989480954980532221, int64 insert_ts=1714528861236612 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569828 396 client-test-util.cc:55] Op INSERT int64 rand_key=-7468845172444954620, int64 link_to=-2293603020743442428, int64 insert_ts=1714528861236646 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569878 396 client-test-util.cc:55] Op INSERT int64 rand_key=-7517626343576633339, int64 link_to=-7136212096608043003, int64 insert_ts=1714528861236675 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569923 396 client-test-util.cc:55] Op INSERT int64 rand_key=-7566548252196732922, int64 link_to=6468063638725197830, int64 insert_ts=1714528861236704 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.569968 396 client-test-util.cc:55] Op INSERT int64 rand_key=-7615188685840121849, int64 link_to=1625595300348887047, int64 insert_ts=1714528861236731 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.570011 396 client-test-util.cc:55] Op INSERT int64 rand_key=-518126252937248752, int64 link_to=2104318721581973520, int64 insert_ts=1714528861237028 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
I20240501 02:02:01.570055 396 client-test-util.cc:55] Op INSERT int64 rand_key=-566766686580637679, int64 link_to=-2738149616794337263, int64 insert_ts=1714528861237075 had status Timed out: Failed to write batch of 35 ops to tablet e1813f5ac1904e7c87f93cac584d3e14 after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: e1813f5ac1904e7c87f93cac584d3e14, num_ops: 35, num_attempts: 343) passed its deadline: Network error: recv got EOF from 127.0.99.2:38843 (error 108)
F20240501 02:02:01.570111 396 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714528921 (unix time) try "date -d @1714528921" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7f571abaf280) from PID 396; stack trace: ***
@ 0x7f5726ec6980 (unknown) at ??:0
@ 0x7f5724b4afb7 gsignal at ??:0
@ 0x7f5724b4c921 abort at ??:0
@ 0x7f5725a57dcd google::LogMessage::Fail() at ??:0
@ 0x7f5725a5bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5725a577cc google::LogMessage::Flush() at ??:0
@ 0x7f5725a58f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f57281897f2 kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x55a29d979236 kudu::client::FlushSessionOrDie() at ??:0
@ 0x55a29d97ccaa kudu::LinkedListTester::LoadLinkedList() at ??:0
@ 0x55a29d9724ca kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at ??:0
@ 0x7f572694e0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f5726942bdb testing::Test::Run() at ??:0
@ 0x7f5726942d9d testing::TestInfo::Run() at ??:0
@ 0x7f5726943377 testing::TestSuite::Run() at ??:0
@ 0x7f572694377c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f572694e60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f5726942e63 testing::UnitTest::Run() at ??:0
@ 0x7f572819b1de RUN_ALL_TESTS() at ??:0
@ 0x7f572819a53d main at ??:0
@ 0x7f5724b2dbf7 __libc_start_main at ??:0
@ 0x55a29d97034a _start at ??:0