[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:49.137403 31305 test_util.cc:268] Using random seed: 109737702
I20240501 14:08:49.144132 31305 linked_list-test.cc:100] Linked List Test Configuration:
I20240501 14:08:49.144207 31305 linked_list-test.cc:101] --------------
I20240501 14:08:49.144222 31305 linked_list-test.cc:102] 50 chains
I20240501 14:08:49.144235 31305 linked_list-test.cc:103] 3 tablets
I20240501 14:08:49.144248 31305 linked_list-test.cc:104] Mutations on
I20240501 14:08:49.144260 31305 linked_list-test.cc:105] --------------
I20240501 14:08:49.144342 31305 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:49.144376 31305 ts_itest-base.cc:113] --------------
I20240501 14:08:49.144392 31305 ts_itest-base.cc:114] 3 tablet servers
I20240501 14:08:49.144404 31305 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:49.144424 31305 ts_itest-base.cc:116] --------------
2024-05-01T14:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:49Z Disabled control of system clock
I20240501 14:08:49.154111 31305 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
/tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.146.126:34185
--webserver_interface=127.30.146.126
--webserver_port=0
--builtin_ntp_servers=127.30.146.84:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.146.126:34185
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240501 14:08:49.270495 31317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.270764 31317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.270819 31317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.274088 31317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:49.274173 31317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.274194 31317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:49.274214 31317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:49.278153 31317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.146.84:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.146.126:34185
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.146.126:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.30.146.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.31317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.279150 31317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.280356 31317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.287982 31322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.288447 31325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.288846 31323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.289268 31317 server_base.cc:1034] running on GCE node
I20240501 14:08:49.289808 31317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.290803 31317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.291945 31317 hybrid_clock.cc:648] HybridClock initialized: now 1714572529291924 us; error 38 us; skew 500 ppm
I20240501 14:08:49.293852 31317 webserver.cc:460] Webserver started at http://127.30.146.126:36615/ using document root <none> and password file <none>
I20240501 14:08:49.294399 31317 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.294458 31317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.294629 31317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.296386 31317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data/instance:
uuid: "5a162316c1214c75af9f5ca1f220eb5e"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.296854 31317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal/instance:
uuid: "5a162316c1214c75af9f5ca1f220eb5e"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.300201 31317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:49.302384 31331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.303390 31317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.303505 31317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal
uuid: "5a162316c1214c75af9f5ca1f220eb5e"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.303592 31317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.317801 31317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.318568 31317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.318730 31317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.325258 31317 rpc_server.cc:307] RPC server started. Bound to: 127.30.146.126:34185
I20240501 14:08:49.325263 31383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.146.126:34185 every 8 connection(s)
I20240501 14:08:49.326290 31317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/data/info.pb
I20240501 14:08:49.328904 31384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.329183 31305 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu as pid 31317
I20240501 14:08:49.329304 31305 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/master-0/wal/instance
I20240501 14:08:49.334518 31384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: Bootstrap starting.
I20240501 14:08:49.336655 31384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.337340 31384 log.cc:824] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:49.338922 31384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: No bootstrap required, opened a new log
I20240501 14:08:49.341575 31384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } }
I20240501 14:08:49.341771 31384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:49.341827 31384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a162316c1214c75af9f5ca1f220eb5e, State: Initialized, Role: FOLLOWER
I20240501 14:08:49.342270 31384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } }
I20240501 14:08:49.342460 31384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:49.342532 31384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:49.342643 31384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:49.343538 31384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } }
I20240501 14:08:49.343878 31384 leader_election.cc:302] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a162316c1214c75af9f5ca1f220eb5e; no voters:
I20240501 14:08:49.344138 31384 leader_election.cc:288] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:49.344775 31389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:49.344974 31389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [term 1 LEADER]: Becoming Leader. State: Replica: 5a162316c1214c75af9f5ca1f220eb5e, State: Running, Role: LEADER
I20240501 14:08:49.345297 31389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } }
I20240501 14:08:49.345347 31384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:49.346305 31390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a162316c1214c75af9f5ca1f220eb5e. Latest consensus state: current_term: 1 leader_uuid: "5a162316c1214c75af9f5ca1f220eb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } } }
I20240501 14:08:49.346412 31390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:49.346776 31391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a162316c1214c75af9f5ca1f220eb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a162316c1214c75af9f5ca1f220eb5e" member_type: VOTER last_known_addr { host: "127.30.146.126" port: 34185 } } }
I20240501 14:08:49.346868 31391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e [sys.catalog]: This master's current role is: LEADER
W20240501 14:08:49.350252 31404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:08:49.350332 31404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:08:49.350431 31398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:49.351187 31398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:49.356271 31398 catalog_manager.cc:1347] Generated new cluster ID: 2a673f737074438f83c80a8db82fdd3a
I20240501 14:08:49.356345 31398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:49.377589 31398 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:49.378903 31398 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:49.394807 31398 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5a162316c1214c75af9f5ca1f220eb5e: Generated new TSK 0
I20240501 14:08:49.395571 31398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:49.407230 31305 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
/tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.146.65:0
--local_ip_for_outbound_sockets=127.30.146.65
--webserver_interface=127.30.146.65
--webserver_port=0
--tserver_master_addrs=127.30.146.126:34185
--builtin_ntp_servers=127.30.146.84:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240501 14:08:49.531296 31408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.531522 31408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.531580 31408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.534754 31408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.534848 31408 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 14:08:49.534863 31408 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 14:08:49.534889 31408 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 14:08:49.534938 31408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.146.65
I20240501 14:08:49.538691 31408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.146.84:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.146.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.30.146.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.30.146.126:34185
--never_fsync=true
--heap_profile_path=/tmp/kudu.31408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.146.65
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.539606 31408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.540760 31408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.549170 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.549430 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.549752 31416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.550426 31408 server_base.cc:1034] running on GCE node
I20240501 14:08:49.550812 31408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.551403 31408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.552546 31408 hybrid_clock.cc:648] HybridClock initialized: now 1714572529552520 us; error 41 us; skew 500 ppm
I20240501 14:08:49.554471 31408 webserver.cc:460] Webserver started at http://127.30.146.65:36461/ using document root <none> and password file <none>
I20240501 14:08:49.555043 31408 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.555104 31408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.555296 31408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.557060 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data/instance:
uuid: "954927af57d74f00b3868b538afc045a"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.557552 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal/instance:
uuid: "954927af57d74f00b3868b538afc045a"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.561010 31408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:49.563249 31422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.564304 31408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.564407 31408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal
uuid: "954927af57d74f00b3868b538afc045a"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.564499 31408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.579226 31408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.579922 31408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.580073 31408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.580605 31408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.581486 31408 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.581532 31408 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.581576 31408 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.581604 31408 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.589735 31408 rpc_server.cc:307] RPC server started. Bound to: 127.30.146.65:38615
I20240501 14:08:49.589758 31535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.146.65:38615 every 8 connection(s)
I20240501 14:08:49.590754 31408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/data/info.pb
I20240501 14:08:49.593407 31305 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu as pid 31408
I20240501 14:08:49.593525 31305 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-0/wal/instance
I20240501 14:08:49.595930 31305 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
/tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.146.66:0
--local_ip_for_outbound_sockets=127.30.146.66
--webserver_interface=127.30.146.66
--webserver_port=0
--tserver_master_addrs=127.30.146.126:34185
--builtin_ntp_servers=127.30.146.84:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 14:08:49.600023 31536 heartbeater.cc:347] Connected to a master server at 127.30.146.126:34185
I20240501 14:08:49.600247 31536 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.600782 31536 heartbeater.cc:511] Master 127.30.146.126:34185 requested a full tablet report, sending...
I20240501 14:08:49.602213 31348 ts_manager.cc:194] Registered new tserver with Master: 954927af57d74f00b3868b538afc045a (127.30.146.65:38615)
I20240501 14:08:49.603372 31348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.146.65:53667
W20240501 14:08:49.701721 31539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.701992 31539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.702072 31539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.705314 31539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.705415 31539 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 14:08:49.705437 31539 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 14:08:49.705466 31539 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 14:08:49.705534 31539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.146.66
I20240501 14:08:49.709367 31539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.146.84:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.146.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.30.146.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.30.146.126:34185
--never_fsync=true
--heap_profile_path=/tmp/kudu.31539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.146.66
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.710337 31539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.711464 31539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.719327 31547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.719664 31545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.719969 31544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.720595 31539 server_base.cc:1034] running on GCE node
I20240501 14:08:49.720974 31539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.721508 31539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.722649 31539 hybrid_clock.cc:648] HybridClock initialized: now 1714572529722636 us; error 27 us; skew 500 ppm
I20240501 14:08:49.724418 31539 webserver.cc:460] Webserver started at http://127.30.146.66:37569/ using document root <none> and password file <none>
I20240501 14:08:49.724942 31539 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.724996 31539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.725155 31539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.726752 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data/instance:
uuid: "36a46c97abd14ff68d8d8465313e9d79"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.727186 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal/instance:
uuid: "36a46c97abd14ff68d8d8465313e9d79"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.730325 31539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:49.732374 31553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.733338 31539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.733436 31539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal
uuid: "36a46c97abd14ff68d8d8465313e9d79"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.733520 31539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.747248 31539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.747939 31539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.748090 31539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.748631 31539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.749536 31539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.749588 31539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.749642 31539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.749665 31539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.757769 31539 rpc_server.cc:307] RPC server started. Bound to: 127.30.146.66:43151
I20240501 14:08:49.757788 31666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.146.66:43151 every 8 connection(s)
I20240501 14:08:49.758744 31539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/data/info.pb
I20240501 14:08:49.760782 31305 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu as pid 31539
I20240501 14:08:49.760881 31305 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-1/wal/instance
I20240501 14:08:49.763191 31305 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
/tmp/dist-test-taskOhKKdc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.146.67:0
--local_ip_for_outbound_sockets=127.30.146.67
--webserver_interface=127.30.146.67
--webserver_port=0
--tserver_master_addrs=127.30.146.126:34185
--builtin_ntp_servers=127.30.146.84:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 14:08:49.768110 31667 heartbeater.cc:347] Connected to a master server at 127.30.146.126:34185
I20240501 14:08:49.768349 31667 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.768955 31667 heartbeater.cc:511] Master 127.30.146.126:34185 requested a full tablet report, sending...
I20240501 14:08:49.769996 31348 ts_manager.cc:194] Registered new tserver with Master: 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151)
I20240501 14:08:49.770742 31348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.146.66:48333
W20240501 14:08:49.873117 31670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.873338 31670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.873395 31670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.876428 31670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.876510 31670 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 14:08:49.876529 31670 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 14:08:49.876551 31670 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 14:08:49.876595 31670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.146.67
I20240501 14:08:49.880290 31670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.146.84:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.146.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.30.146.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.30.146.126:34185
--never_fsync=true
--heap_profile_path=/tmp/kudu.31670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.146.67
--log_dir=/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.881176 31670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.882285 31670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.890522 31678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.890944 31676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.891317 31675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.891722 31670 server_base.cc:1034] running on GCE node
I20240501 14:08:49.892108 31670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.892697 31670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.893847 31670 hybrid_clock.cc:648] HybridClock initialized: now 1714572529893831 us; error 31 us; skew 500 ppm
I20240501 14:08:49.895812 31670 webserver.cc:460] Webserver started at http://127.30.146.67:37317/ using document root <none> and password file <none>
I20240501 14:08:49.896395 31670 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.896452 31670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.896627 31670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.898325 31670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data/instance:
uuid: "aa5fb08274634c9198b5ca985e3239b7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.898778 31670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal/instance:
uuid: "aa5fb08274634c9198b5ca985e3239b7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.902000 31670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 14:08:49.904096 31684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.905058 31670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.905170 31670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal
uuid: "aa5fb08274634c9198b5ca985e3239b7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-8kl2"
I20240501 14:08:49.905247 31670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.928992 31670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.929684 31670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.929845 31670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.930423 31670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.931352 31670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.931398 31670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.931453 31670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.931480 31670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.939719 31670 rpc_server.cc:307] RPC server started. Bound to: 127.30.146.67:39395
I20240501 14:08:49.939738 31797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.146.67:39395 every 8 connection(s)
I20240501 14:08:49.940694 31670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/data/info.pb
I20240501 14:08:49.949276 31798 heartbeater.cc:347] Connected to a master server at 127.30.146.126:34185
I20240501 14:08:49.949478 31798 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.949874 31305 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOhKKdc/build/debug/bin/kudu as pid 31670
I20240501 14:08:49.949993 31305 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOhKKdc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714572529126593-31305-0/linked-list-cluster/ts-2/wal/instance
I20240501 14:08:49.950078 31798 heartbeater.cc:511] Master 127.30.146.126:34185 requested a full tablet report, sending...
I20240501 14:08:49.951236 31347 ts_manager.cc:194] Registered new tserver with Master: aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:49.951884 31347 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.146.67:42101
I20240501 14:08:49.952461 31305 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 14:08:49.963721 31347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43514:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240501 14:08:49.966610 31347 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:08:49.984210 31470 tablet_service.cc:1467] Processing CreateTablet for tablet b66a5f961f61423fa452a4491bbabc67 (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 14:08:49.984325 31732 tablet_service.cc:1467] Processing CreateTablet for tablet b66a5f961f61423fa452a4491bbabc67 (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 14:08:49.984210 31600 tablet_service.cc:1467] Processing CreateTablet for tablet 462974e1e7cc439b87d95a99c93258bf (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 14:08:49.984961 31601 tablet_service.cc:1467] Processing CreateTablet for tablet b66a5f961f61423fa452a4491bbabc67 (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 14:08:49.984212 31731 tablet_service.cc:1467] Processing CreateTablet for tablet 462974e1e7cc439b87d95a99c93258bf (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 14:08:49.984212 31730 tablet_service.cc:1467] Processing CreateTablet for tablet 61752feefd0e4bac8dd22753f55ff1cc (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 14:08:49.985463 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66a5f961f61423fa452a4491bbabc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.985463 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462974e1e7cc439b87d95a99c93258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.985466 31731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462974e1e7cc439b87d95a99c93258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.985812 31730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61752feefd0e4bac8dd22753f55ff1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.984721 31599 tablet_service.cc:1467] Processing CreateTablet for tablet 61752feefd0e4bac8dd22753f55ff1cc (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 14:08:49.986843 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66a5f961f61423fa452a4491bbabc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.986887 31599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61752feefd0e4bac8dd22753f55ff1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.988060 31469 tablet_service.cc:1467] Processing CreateTablet for tablet 462974e1e7cc439b87d95a99c93258bf (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 14:08:49.988328 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462974e1e7cc439b87d95a99c93258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.992033 31468 tablet_service.cc:1467] Processing CreateTablet for tablet 61752feefd0e4bac8dd22753f55ff1cc (DEFAULT_TABLE table=TestTable [id=63455c76f9534d9b962904f009b1a06a]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 14:08:49.992307 31468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61752feefd0e4bac8dd22753f55ff1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.993042 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66a5f961f61423fa452a4491bbabc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:49.993767 31816 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Bootstrap starting.
I20240501 14:08:49.994457 31815 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: Bootstrap starting.
I20240501 14:08:49.995692 31817 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: Bootstrap starting.
I20240501 14:08:49.995942 31816 tablet_bootstrap.cc:654] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.996470 31815 tablet_bootstrap.cc:654] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.997043 31816 log.cc:824] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:49.997387 31817 tablet_bootstrap.cc:654] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:49.997448 31815 log.cc:824] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:49.998340 31817 log.cc:824] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:49.998844 31816 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: No bootstrap required, opened a new log
I20240501 14:08:49.999030 31816 ts_tablet_manager.cc:1367] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:08:49.999085 31815 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: No bootstrap required, opened a new log
I20240501 14:08:49.999248 31815 ts_tablet_manager.cc:1367] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:49.999650 31817 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: No bootstrap required, opened a new log
I20240501 14:08:49.999796 31817 ts_tablet_manager.cc:1367] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:08:50.002154 31817 raft_consensus.cc:348] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.002166 31816 raft_consensus.cc:348] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.002171 31815 raft_consensus.cc:348] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.002334 31817 raft_consensus.cc:374] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.002349 31816 raft_consensus.cc:374] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.002349 31815 raft_consensus.cc:374] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.002384 31817 raft_consensus.cc:729] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.002394 31816 raft_consensus.cc:729] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954927af57d74f00b3868b538afc045a, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.002401 31815 raft_consensus.cc:729] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5fb08274634c9198b5ca985e3239b7, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.002794 31817 consensus_queue.cc:260] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.002805 31816 consensus_queue.cc:260] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.002805 31815 consensus_queue.cc:260] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.003700 31817 ts_tablet_manager.cc:1398] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240501 14:08:50.003703 31815 ts_tablet_manager.cc:1398] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:08:50.003703 31816 ts_tablet_manager.cc:1398] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:50.004105 31817 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: Bootstrap starting.
I20240501 14:08:50.004098 31536 heartbeater.cc:503] Master 127.30.146.126:34185 was elected leader, sending a full tablet report...
I20240501 14:08:50.004103 31798 heartbeater.cc:503] Master 127.30.146.126:34185 was elected leader, sending a full tablet report...
I20240501 14:08:50.004154 31667 heartbeater.cc:503] Master 127.30.146.126:34185 was elected leader, sending a full tablet report...
I20240501 14:08:50.005132 31817 tablet_bootstrap.cc:654] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.005136 31815 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7: Bootstrap starting.
I20240501 14:08:50.006096 31815 tablet_bootstrap.cc:654] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.006263 31817 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: No bootstrap required, opened a new log
I20240501 14:08:50.006332 31817 ts_tablet_manager.cc:1367] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:50.006791 31816 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a: Bootstrap starting.
I20240501 14:08:50.006875 31817 raft_consensus.cc:348] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.006982 31817 raft_consensus.cc:374] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.007019 31817 raft_consensus.cc:729] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.007146 31817 consensus_queue.cc:260] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.007400 31817 ts_tablet_manager.cc:1398] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:50.007577 31817 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79: Bootstrap starting.
I20240501 14:08:50.008020 31816 tablet_bootstrap.cc:654] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.008481 31817 tablet_bootstrap.cc:654] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79: Neither blocks nor log segments found. Creating new log.
W20240501 14:08:50.009575 31668 tablet.cc:2369] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:50.009688 31816 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a: No bootstrap required, opened a new log
I20240501 14:08:50.009768 31816 ts_tablet_manager.cc:1367] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:50.010327 31816 raft_consensus.cc:348] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.010466 31816 raft_consensus.cc:374] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.010524 31816 raft_consensus.cc:729] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954927af57d74f00b3868b538afc045a, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.010669 31816 consensus_queue.cc:260] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.010962 31816 ts_tablet_manager.cc:1398] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:08:50.011153 31816 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a: Bootstrap starting.
I20240501 14:08:50.011654 31817 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79: No bootstrap required, opened a new log
I20240501 14:08:50.011781 31817 ts_tablet_manager.cc:1367] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:08:50.012220 31816 tablet_bootstrap.cc:654] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.012411 31815 tablet_bootstrap.cc:492] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7: No bootstrap required, opened a new log
I20240501 14:08:50.012507 31815 ts_tablet_manager.cc:1367] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:08:50.012531 31817 raft_consensus.cc:348] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.012634 31817 raft_consensus.cc:374] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.012670 31817 raft_consensus.cc:729] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.012809 31817 consensus_queue.cc:260] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.013083 31817 ts_tablet_manager.cc:1398] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:50.013711 31815 raft_consensus.cc:348] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.013839 31815 raft_consensus.cc:374] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.013883 31815 raft_consensus.cc:729] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5fb08274634c9198b5ca985e3239b7, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.013934 31816 tablet_bootstrap.cc:492] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a: No bootstrap required, opened a new log
I20240501 14:08:50.014016 31816 ts_tablet_manager.cc:1367] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:50.014034 31815 consensus_queue.cc:260] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.014396 31816 raft_consensus.cc:348] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.014488 31816 raft_consensus.cc:374] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.014516 31816 raft_consensus.cc:729] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954927af57d74f00b3868b538afc045a, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.014655 31816 consensus_queue.cc:260] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.014793 31815 ts_tablet_manager.cc:1398] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:50.014911 31816 ts_tablet_manager.cc:1398] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:50.014966 31815 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: Bootstrap starting.
I20240501 14:08:50.016036 31815 tablet_bootstrap.cc:654] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.017094 31815 tablet_bootstrap.cc:492] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: No bootstrap required, opened a new log
I20240501 14:08:50.017171 31815 ts_tablet_manager.cc:1367] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:50.017545 31815 raft_consensus.cc:348] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.017633 31815 raft_consensus.cc:374] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.017665 31815 raft_consensus.cc:729] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5fb08274634c9198b5ca985e3239b7, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.017783 31815 consensus_queue.cc:260] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.018023 31815 ts_tablet_manager.cc:1398] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 14:08:50.091814 31537 tablet.cc:2369] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:50.186597 31822 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:50.186815 31822 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.187945 31822 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151)
W20240501 14:08:50.191710 31799 tablet.cc:2369] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:50.192440 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a46c97abd14ff68d8d8465313e9d79" is_pre_election: true
I20240501 14:08:50.192478 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a" is_pre_election: true
I20240501 14:08:50.192720 31490 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 0.
I20240501 14:08:50.192721 31621 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 0.
I20240501 14:08:50.193126 31687 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 954927af57d74f00b3868b538afc045a, aa5fb08274634c9198b5ca985e3239b7; no voters:
I20240501 14:08:50.193392 31822 raft_consensus.cc:2781] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:50.193459 31822 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:50.193488 31822 raft_consensus.cc:3037] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.194324 31822 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.194653 31822 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 election: Requested vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151)
I20240501 14:08:50.194957 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a"
I20240501 14:08:50.195001 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a46c97abd14ff68d8d8465313e9d79"
I20240501 14:08:50.195083 31490 raft_consensus.cc:3037] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.195114 31621 raft_consensus.cc:3037] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.195971 31621 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 1.
I20240501 14:08:50.196240 31688 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7; no voters:
I20240501 14:08:50.196290 31490 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 1.
I20240501 14:08:50.196380 31822 raft_consensus.cc:2781] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.196599 31822 raft_consensus.cc:686] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [term 1 LEADER]: Becoming Leader. State: Replica: aa5fb08274634c9198b5ca985e3239b7, State: Running, Role: LEADER
I20240501 14:08:50.196918 31822 consensus_queue.cc:237] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.200034 31347 catalog_manager.cc:5484] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67). New cstate: current_term: 1 leader_uuid: "aa5fb08274634c9198b5ca985e3239b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:50.212620 31821 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:50.212764 31821 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.213831 31821 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:50.217396 31752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462974e1e7cc439b87d95a99c93258bf" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5fb08274634c9198b5ca985e3239b7" is_pre_election: true
I20240501 14:08:50.217432 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462974e1e7cc439b87d95a99c93258bf" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a" is_pre_election: true
I20240501 14:08:50.217586 31490 raft_consensus.cc:2454] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 0.
I20240501 14:08:50.217638 31752 raft_consensus.cc:2454] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 0.
I20240501 14:08:50.217978 31556 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters:
I20240501 14:08:50.218222 31821 raft_consensus.cc:2781] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:50.218281 31821 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:50.218320 31821 raft_consensus.cc:3037] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.219103 31821 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.219440 31821 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 1 election: Requested vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:50.219811 31752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462974e1e7cc439b87d95a99c93258bf" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5fb08274634c9198b5ca985e3239b7"
I20240501 14:08:50.219811 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462974e1e7cc439b87d95a99c93258bf" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a"
I20240501 14:08:50.219926 31752 raft_consensus.cc:3037] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.219928 31490 raft_consensus.cc:3037] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.220806 31752 raft_consensus.cc:2454] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 1.
I20240501 14:08:50.220806 31490 raft_consensus.cc:2454] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 1.
I20240501 14:08:50.221082 31556 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters:
I20240501 14:08:50.221263 31821 raft_consensus.cc:2781] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.221522 31821 raft_consensus.cc:686] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [term 1 LEADER]: Becoming Leader. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Running, Role: LEADER
I20240501 14:08:50.221818 31821 consensus_queue.cc:237] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.224685 31347 catalog_manager.cc:5484] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66). New cstate: current_term: 1 leader_uuid: "36a46c97abd14ff68d8d8465313e9d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:50.292363 31822 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:50.292484 31822 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.292907 31822 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), 954927af57d74f00b3868b538afc045a (127.30.146.65:38615)
I20240501 14:08:50.293376 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a46c97abd14ff68d8d8465313e9d79" is_pre_election: true
I20240501 14:08:50.293406 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a" is_pre_election: true
I20240501 14:08:50.293502 31621 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 0.
I20240501 14:08:50.293556 31490 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 0.
I20240501 14:08:50.293773 31688 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7; no voters:
I20240501 14:08:50.293959 31822 raft_consensus.cc:2781] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:50.294016 31822 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:50.294041 31822 raft_consensus.cc:3037] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.294783 31822 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.295094 31822 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 election: Requested vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), 954927af57d74f00b3868b538afc045a (127.30.146.65:38615)
I20240501 14:08:50.295418 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a"
I20240501 14:08:50.295423 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "aa5fb08274634c9198b5ca985e3239b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a46c97abd14ff68d8d8465313e9d79"
I20240501 14:08:50.295521 31490 raft_consensus.cc:3037] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.295536 31621 raft_consensus.cc:3037] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.296280 31621 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 1.
I20240501 14:08:50.296294 31490 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa5fb08274634c9198b5ca985e3239b7 in term 1.
I20240501 14:08:50.296532 31688 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7; no voters:
I20240501 14:08:50.296727 31822 raft_consensus.cc:2781] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.296801 31822 raft_consensus.cc:686] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [term 1 LEADER]: Becoming Leader. State: Replica: aa5fb08274634c9198b5ca985e3239b7, State: Running, Role: LEADER
I20240501 14:08:50.296942 31822 consensus_queue.cc:237] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:50.298486 31347 catalog_manager.cc:5484] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67). New cstate: current_term: 1 leader_uuid: "aa5fb08274634c9198b5ca985e3239b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:50.347115 31305 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 14:08:50.348907 31305 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 954927af57d74f00b3868b538afc045a to finish bootstrapping
I20240501 14:08:50.354329 31305 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 36a46c97abd14ff68d8d8465313e9d79 to finish bootstrapping
I20240501 14:08:50.358880 31305 ts_itest-base.cc:233] Waiting for 3 tablets on tserver aa5fb08274634c9198b5ca985e3239b7 to finish bootstrapping
I20240501 14:08:50.364106 31838 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.30.146.66:37569/startup
http://127.30.146.126:36615/dump-entities
http://127.30.146.67:37317/stacks
http://127.30.146.126:36615/stacks
http://127.30.146.67:37317/mem-trackers
http://127.30.146.65:36461/maintenance-manager
http://127.30.146.65:36461/threadz
http://127.30.146.66:37569/maintenance-manager
http://127.30.146.65:36461/stacks
http://127.30.146.65:36461/scans
http://127.30.146.126:36615/tablet-servers
http://127.30.146.126:36615/threadz
http://127.30.146.126:36615/ipki-ca-cert
http://127.30.146.66:37569/threadz
http://127.30.146.66:37569/tablets
http://127.30.146.66:37569/metrics
http://127.30.146.65:36461/startup
http://127.30.146.67:37317/transactions?tablet_id=61752feefd0e4bac8dd22753f55ff1cc
http://127.30.146.67:37317/tablets
http://127.30.146.67:37317/scans
http://127.30.146.66:37569/scans
http://127.30.146.65:36461/transactions?tablet_id=61752feefd0e4bac8dd22753f55ff1cc
http://127.30.146.67:37317/threadz
http://127.30.146.66:37569/threadz?group=all
http://127.30.146.126:36615/ipki-ca-cert-der
http://127.30.146.126:36615/threadz?group=all
http://127.30.146.66:37569/transactions?tablet_id=61752feefd0e4bac8dd22753f55ff1cc
http://127.30.146.65:36461/threadz?group=all
http://127.30.146.67:37317/metrics
http://127.30.146.66:37569/mem-trackers
http://127.30.146.65:36461/tablets
http://127.30.146.126:36615/ipki-ca-cert-pem
http://127.30.146.65:36461/metrics
http://127.30.146.126:36615/masters
http://127.30.146.126:36615/startup
http://127.30.146.126:36615/mem-trackers
http://127.30.146.126:36615/metrics
http://127.30.146.65:36461/mem-trackers
http://127.30.146.67:37317/maintenance-manager
http://127.30.146.67:37317/threadz?group=all
http://127.30.146.126:36615/tables
http://127.30.146.66:37569/stacks
http://127.30.146.67:37317/startup
I20240501 14:08:50.367197 31305 linked_list-test-util.h:401] Running for: 30.000s
I20240501 14:08:50.367261 31305 linked_list-test-util.h:402] Sampling every 10000000 us
I20240501 14:08:50.367277 31305 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240501 14:08:50.386798 31752 raft_consensus.cc:1261] T 462974e1e7cc439b87d95a99c93258bf P aa5fb08274634c9198b5ca985e3239b7 [term 1 FOLLOWER]: Refusing update from remote peer 36a46c97abd14ff68d8d8465313e9d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.387545 31834 consensus_queue.cc:1035] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.387894 31621 raft_consensus.cc:1261] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Refusing update from remote peer aa5fb08274634c9198b5ca985e3239b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.387905 31490 raft_consensus.cc:1261] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Refusing update from remote peer aa5fb08274634c9198b5ca985e3239b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.388324 31620 raft_consensus.cc:1261] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Refusing update from remote peer aa5fb08274634c9198b5ca985e3239b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.388515 31831 consensus_queue.cc:1035] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.388844 31822 consensus_queue.cc:1035] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.388868 31831 consensus_queue.cc:1035] T b66a5f961f61423fa452a4491bbabc67 P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.390307 31490 raft_consensus.cc:1261] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Refusing update from remote peer 36a46c97abd14ff68d8d8465313e9d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.391686 31821 consensus_queue.cc:1035] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.392452 31490 raft_consensus.cc:1261] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Refusing update from remote peer aa5fb08274634c9198b5ca985e3239b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:50.393215 31822 consensus_queue.cc:1035] T 61752feefd0e4bac8dd22753f55ff1cc P aa5fb08274634c9198b5ca985e3239b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:50.397400 31853 mvcc.cc:205] Tried to move back new op lower bound from 7022889084456513536 to 7022889083790872576. Current Snapshot: MvccSnapshot[applied={T|T < 7022889084456513536}]
I20240501 14:08:50.397621 31847 mvcc.cc:205] Tried to move back new op lower bound from 7022889084456513536 to 7022889083790872576. Current Snapshot: MvccSnapshot[applied={T|T < 7022889084456513536}]
I20240501 14:08:50.401435 31851 mvcc.cc:205] Tried to move back new op lower bound from 7022889084461285376 to 7022889084097425408. Current Snapshot: MvccSnapshot[applied={T|T < 7022889084461285376}]
I20240501 14:08:51.968577 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:52.007242 31842 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:52.008200 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:53.598309 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:08:53.699194 31799 maintenance_manager.cc:419] P aa5fb08274634c9198b5ca985e3239b7: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:08:54.018638 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:08:54.311544 31689 maintenance_manager.cc:643] P aa5fb08274634c9198b5ca985e3239b7: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.610s user 0.274s sys 0.004s Metrics: {"bytes_written":151122,"cfile_init":1,"compiler_manager_pool.queue_time_us":20203,"dirs.queue_time_us":37256,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":43922,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10567,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8291,"thread_start_us":180,"threads_started":2}
I20240501 14:08:54.313905 31799 maintenance_manager.cc:419] P aa5fb08274634c9198b5ca985e3239b7: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 142822 bytes on disk
I20240501 14:08:54.315266 31689 maintenance_manager.cc:643] P aa5fb08274634c9198b5ca985e3239b7: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 14:08:54.410032 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.811s user 0.246s sys 0.008s Metrics: {"bytes_written":147033,"cfile_init":1,"compiler_manager_pool.queue_time_us":41324,"compiler_manager_pool.run_cpu_time_us":245913,"compiler_manager_pool.run_wall_time_us":509452,"dirs.queue_time_us":13460,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4239,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8063,"spinlock_wait_cycles":134912,"thread_start_us":207,"threads_started":2}
I20240501 14:08:54.411473 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 140277 bytes on disk
I20240501 14:08:54.412524 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 14:08:54.846110 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.827s user 0.273s sys 0.017s Metrics: {"bytes_written":6755,"cfile_init":1,"compiler_manager_pool.queue_time_us":19194,"compiler_manager_pool.run_cpu_time_us":254599,"compiler_manager_pool.run_wall_time_us":667078,"delete_count":0,"dirs.queue_time_us":7477,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":51254,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6420,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8888,"thread_start_us":177,"threads_started":2,"update_count":1025}
I20240501 14:08:54.849185 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 148958 bytes on disk
I20240501 14:08:54.852068 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:08:54.852689 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67): perf score=0.048303
I20240501 14:08:54.921934 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.066s user 0.031s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":8027,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":40,"lbm_read_time_us":230,"lbm_reads_lt_1ms":20,"lbm_write_time_us":258,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":86,"threads_started":1,"update_count":1025}
I20240501 14:08:55.004571 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 14:08:55.038712 31842 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fe115775dcd google::LogMessage::Fail() at ??:0
@ 0x7fe115779b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe1157757cc google::LogMessage::Flush() at ??:0
@ 0x7fe115776f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe116b575b1 kudu::operator-() at ??:0
@ 0x7fe118c3afd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fe118c2405b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fe118c23f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fe118bfa9bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fe119826861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fe11982593a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fe11982e149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fe1146f0d09 std::function<>::operator()() at ??:0
@ 0x7fe1146eb107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fe1146eaa38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fe1146ea123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fe114712b89 handle_request at ??:0
@ 0x7fe11471526b worker_thread at ??:0
@ 0x7fe1173f46db start_thread at ??:0
@ 0x7fe11492c71f clone at ??:0
W20240501 14:08:55.259312 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: recv got EOF from 127.30.146.67:39395 (error 108)
I20240501 14:08:55.260325 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:08:55.267535 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.275550 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.290808 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.301429 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.323515 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.336355 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.362709 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.376479 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.407939 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.424799 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.464350 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.485322 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.512550 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.519088 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.520162 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.520740 31554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:08:55.521610 31554 consensus_peers.cc:485] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:55.526567 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.539449 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.541843 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.550268 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.551419 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.553956 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.574537 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.574537 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.579205 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.584604 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.588085 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.612128 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.615483 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.625902 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.636121 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.637391 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.664736 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.664736 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.676308 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.684581 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.696475 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.722194 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.725312 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.747150 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.751727 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.760438 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.795218 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.804651 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.819125 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.819829 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.856849 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.869323 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.884615 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.884637 31580 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.895539 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.954319 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:55.957563 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.975167 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.985108 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:55.994829 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.008191 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.048856 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.077267 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.079241 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.112097 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.149138 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.157737 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.183216 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.186944 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.198684 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.220244 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.260025 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:08:56.263259 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.301208 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.303797 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.334517 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.340119 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.384776 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
I20240501 14:08:56.407722 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:08:56.428436 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.429951 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
I20240501 14:08:56.431895 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:08:56.465736 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
W20240501 14:08:56.508836 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.513322 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
I20240501 14:08:56.534631 31924 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa5fb08274634c9198b5ca985e3239b7)
I20240501 14:08:56.534765 31924 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.535184 31924 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:56.535606 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 315 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a" is_pre_election: true
I20240501 14:08:56.535782 31489 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 1.
I20240501 14:08:56.536155 31556 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters:
I20240501 14:08:56.536327 31924 raft_consensus.cc:2781] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:08:56.536377 31924 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa5fb08274634c9198b5ca985e3239b7)
I20240501 14:08:56.536422 31924 raft_consensus.cc:3037] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:08:56.536494 31554 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:08:56.544667 31927 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa5fb08274634c9198b5ca985e3239b7)
I20240501 14:08:56.544799 31927 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.546018 31927 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:56.546815 31871 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa5fb08274634c9198b5ca985e3239b7)
W20240501 14:08:56.546864 31423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:08:56.546917 31871 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.547312 31871 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:56.547660 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 323 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a" is_pre_election: true
I20240501 14:08:56.547809 31489 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 1.
I20240501 14:08:56.548110 31556 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters:
I20240501 14:08:56.548280 31871 raft_consensus.cc:2781] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 14:08:56.548269 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:08:56.548326 31871 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa5fb08274634c9198b5ca985e3239b7)
I20240501 14:08:56.548359 31871 raft_consensus.cc:3037] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:08:56.548766 31554 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:08:56.551045 31620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "954927af57d74f00b3868b538afc045a" candidate_term: 2 candidate_status { last_received { term: 1 index: 323 } } ignore_live_leader: false dest_uuid: "36a46c97abd14ff68d8d8465313e9d79" is_pre_election: true
W20240501 14:08:56.557130 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:08:56.567870 31924 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.568420 31924 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: Requested vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:56.568825 31871 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.569005 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61752feefd0e4bac8dd22753f55ff1cc" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 315 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a"
I20240501 14:08:56.569131 31489 raft_consensus.cc:3037] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:56.569236 31871 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: Requested vote from peers 954927af57d74f00b3868b538afc045a (127.30.146.65:38615), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:08:56.569422 31620 raft_consensus.cc:2379] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 954927af57d74f00b3868b538afc045a in current term 2: Already voted for candidate 36a46c97abd14ff68d8d8465313e9d79 in this term.
I20240501 14:08:56.570142 31489 raft_consensus.cc:2454] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 2.
I20240501 14:08:56.570708 31556 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters:
I20240501 14:08:56.570812 31426 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:08:56.571003 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66a5f961f61423fa452a4491bbabc67" candidate_uuid: "36a46c97abd14ff68d8d8465313e9d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 323 } } ignore_live_leader: false dest_uuid: "954927af57d74f00b3868b538afc045a"
I20240501 14:08:56.571113 31489 raft_consensus.cc:3037] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:56.571453 31871 raft_consensus.cc:2781] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:56.571547 31871 raft_consensus.cc:686] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [term 2 LEADER]: Becoming Leader. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Running, Role: LEADER
I20240501 14:08:56.571698 31871 consensus_queue.cc:237] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 315, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
W20240501 14:08:56.572899 31554 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:08:56.573050 31554 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:08:56.573596 31345 catalog_manager.cc:5484] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 reported cstate change: term changed from 1 to 2, leader changed from aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67) to 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66). New cstate: current_term: 2 leader_uuid: "36a46c97abd14ff68d8d8465313e9d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:56.584633 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:08:56.601752 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:08:56.638711 31487 raft_consensus.cc:1261] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Refusing update from remote peer 36a46c97abd14ff68d8d8465313e9d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20240501 14:08:56.639214 31924 consensus_queue.cc:1035] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
W20240501 14:08:56.639808 31554 consensus_peers.cc:485] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:56.648926 31581 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44504: Illegal state: replica 36a46c97abd14ff68d8d8465313e9d79 is not leader of this config: current role FOLLOWER
I20240501 14:08:56.664736 31489 raft_consensus.cc:2454] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a46c97abd14ff68d8d8465313e9d79 in term 2.
I20240501 14:08:56.664943 31927 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:08:56.665488 31556 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36a46c97abd14ff68d8d8465313e9d79, 954927af57d74f00b3868b538afc045a; no voters: aa5fb08274634c9198b5ca985e3239b7
I20240501 14:08:56.666064 31871 raft_consensus.cc:2781] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:56.666265 31871 raft_consensus.cc:686] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [term 2 LEADER]: Becoming Leader. State: Replica: 36a46c97abd14ff68d8d8465313e9d79, State: Running, Role: LEADER
I20240501 14:08:56.666407 31871 consensus_queue.cc:237] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 323, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:08:56.668671 31346 catalog_manager.cc:5484] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 reported cstate change: term changed from 1 to 2, leader changed from aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67) to 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66). New cstate: current_term: 2 leader_uuid: "36a46c97abd14ff68d8d8465313e9d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.671877 31489 raft_consensus.cc:1261] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Refusing update from remote peer 36a46c97abd14ff68d8d8465313e9d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
W20240501 14:08:56.671952 31554 consensus_peers.cc:485] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:56.672796 31871 consensus_queue.cc:1035] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
W20240501 14:08:56.700496 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:08:57.913849 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:57.937561 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:08:58.122413 31554 consensus_peers.cc:485] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:58.921219 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
W20240501 14:08:59.078966 31554 consensus_peers.cc:485] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:59.143818 31554 consensus_peers.cc:485] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:59.171934 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.250s user 0.202s sys 0.000s Metrics: {"bytes_written":147115,"cfile_init":1,"dirs.queue_time_us":443,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":4202,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1709,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7963,"thread_start_us":74,"threads_started":1}
I20240501 14:08:59.173215 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 134053 bytes on disk
I20240501 14:08:59.174001 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 14:08:59.174520 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.014819
I20240501 14:08:59.305987 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:59.337242 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:08:59.429236 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:08:59.539403 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.365s user 0.349s sys 0.004s Metrics: {"bytes_written":294646,"cfile_cache_hit":12,"cfile_cache_hit_bytes":95178,"cfile_cache_miss":56,"cfile_cache_miss_bytes":957582,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":749,"drs_written":1,"lbm_read_time_us":1950,"lbm_reads_lt_1ms":84,"lbm_write_time_us":2376,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":508731,"rows_written":16026,"thread_start_us":388,"threads_started":6}
I20240501 14:08:59.540555 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 285084 bytes on disk
I20240501 14:08:59.540968 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240501 14:08:59.650444 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.221s user 0.200s sys 0.000s Metrics: {"bytes_written":153030,"cfile_init":1,"dirs.queue_time_us":1107,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1317,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8337,"thread_start_us":100,"threads_started":1}
I20240501 14:08:59.651628 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 131451 bytes on disk
I20240501 14:08:59.652194 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 14:08:59.652802 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.014809
I20240501 14:09:00.380679 31305 linked_list-test-util.h:417] Sample at HT timestamp: 7022889125361057792 Inserted count: 28200
W20240501 14:09:00.559669 31554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:00.560453 31554 consensus_peers.cc:485] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:00.658344 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 1.005s user 0.399s sys 0.003s Metrics: {"bytes_written":12913,"cfile_cache_hit":14,"cfile_cache_hit_bytes":153726,"cfile_cache_miss":60,"cfile_cache_miss_bytes":960592,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":774,"drs_written":1,"lbm_read_time_us":1892,"lbm_reads_lt_1ms":92,"lbm_write_time_us":3941,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":56,"num_input_rowsets":2,"peak_mem_usage":530956,"reinsert_count":0,"rows_written":17225,"update_count":1999}
I20240501 14:09:00.659161 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67): perf score=0.047353
I20240501 14:09:00.715171 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.053s user 0.038s sys 0.004s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15512,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":333,"lbm_reads_lt_1ms":25,"lbm_write_time_us":234,"lbm_writes_lt_1ms":9,"reinsert_count":0,"update_count":1999}
I20240501 14:09:00.720443 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 298152 bytes on disk
I20240501 14:09:00.721305 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 14:09:01.057658 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:01.088423 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:01.380676 31554 consensus_peers.cc:485] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:01.536852 31554 consensus_peers.cc:485] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:02.300482 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:09:02.548590 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.248s user 0.200s sys 0.000s Metrics: {"bytes_written":153186,"cfile_init":1,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":23815,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1336,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8415,"thread_start_us":73,"threads_started":1}
I20240501 14:09:02.549261 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.014706
I20240501 14:09:02.717643 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:02.725947 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling FlushMRSOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.000000
I20240501 14:09:02.754320 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:03.021926 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: FlushMRSOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.296s user 0.236s sys 0.004s Metrics: {"bytes_written":17296,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":14041,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":764,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1566,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8312,"thread_start_us":94,"threads_started":1,"update_count":2693}
I20240501 14:09:03.022840 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67): perf score=1.014692
W20240501 14:09:03.188822 31554 consensus_peers.cc:485] T 462974e1e7cc439b87d95a99c93258bf P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:03.723711 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 1.174s user 0.564s sys 0.000s Metrics: {"bytes_written":10558,"cfile_cache_hit":13,"cfile_cache_hit_bytes":234521,"cfile_cache_miss":71,"cfile_cache_miss_bytes":1377385,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":1221,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":3600,"drs_written":1,"lbm_read_time_us":2889,"lbm_reads_lt_1ms":103,"lbm_write_time_us":5391,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":66,"num_input_rowsets":2,"peak_mem_usage":764869,"reinsert_count":0,"rows_written":24441,"thread_start_us":449,"threads_started":8,"update_count":1630}
I20240501 14:09:03.724592 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 438955 bytes on disk
I20240501 14:09:03.725057 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 14:09:03.725597 31537 maintenance_manager.cc:419] P 954927af57d74f00b3868b538afc045a: Scheduling MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67): perf score=0.027439
I20240501 14:09:03.784408 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: CompactRowSetsOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.761s user 0.572s sys 0.004s Metrics: {"bytes_written":460279,"cfile_cache_hit":14,"cfile_cache_hit_bytes":198416,"cfile_cache_miss":82,"cfile_cache_miss_bytes":1507642,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":12417,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":2519,"drs_written":1,"lbm_read_time_us":2788,"lbm_reads_lt_1ms":122,"lbm_write_time_us":5788,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":844672,"rows_written":25537,"thread_start_us":455,"threads_started":8}
I20240501 14:09:03.785092 31668 maintenance_manager.cc:419] P 36a46c97abd14ff68d8d8465313e9d79: Scheduling UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67): 473207 bytes on disk
I20240501 14:09:03.785502 31558 maintenance_manager.cc:643] P 36a46c97abd14ff68d8d8465313e9d79: UndoDeltaBlockGCOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 14:09:03.804863 31427 maintenance_manager.cc:643] P 954927af57d74f00b3868b538afc045a: MajorDeltaCompactionOp(b66a5f961f61423fa452a4491bbabc67) complete. Timing: real 0.079s user 0.054s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":13697,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":172,"lbm_read_time_us":248,"lbm_reads_lt_1ms":20,"lbm_write_time_us":258,"lbm_writes_lt_1ms":8,"mutex_wait_us":20,"reinsert_count":0,"spinlock_wait_cycles":2944,"update_count":1630}
W20240501 14:09:03.975842 31554 consensus_peers.cc:485] T b66a5f961f61423fa452a4491bbabc67 P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:04.079277 31554 consensus_peers.cc:485] T 61752feefd0e4bac8dd22753f55ff1cc P 36a46c97abd14ff68d8d8465313e9d79 -> Peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Couldn't send request to peer aa5fb08274634c9198b5ca985e3239b7. Status: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:04.627785 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:04.658162 31839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 14:09:04.683805 31839 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fe850b92dcd google::LogMessage::Fail() at ??:0
@ 0x7fe850b96b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe850b927cc google::LogMessage::Flush() at ??:0
@ 0x7fe850b93f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe851f745b1 kudu::operator-() at ??:0
@ 0x7fe854057fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fe85404105b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fe854040f90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fe8540179bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fe854c43861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fe854c4293a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fe854c4b149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fe84fb0dd09 std::function<>::operator()() at ??:0
@ 0x7fe84fb08107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fe84fb07a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fe84fb07123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fe84fb2fb89 handle_request at ??:0
@ 0x7fe84fb3226b worker_thread at ??:0
@ 0x7fe8528116db start_thread at ??:0
@ 0x7fe84fd4971f clone at ??:0
W20240501 14:09:04.887899 31804 connection.cc:535] client connection to 127.30.146.66:43151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:04.888038 31804 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:04.892431 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.892484 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.893289 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.893536 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.912873 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.918661 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.929436 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.943907 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.957536 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.978013 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:04.983255 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.010865 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.032629 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.032629 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.069835 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.088629 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.128417 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.153738 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.154771 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.228384 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.254592 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.312167 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.342324 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.361080 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.400254 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.436805 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.475214 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.502669 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.545287 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:05.601189 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:05.614681 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.655997 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.699035 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.730432 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.782661 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.855944 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.891999 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:05.912376 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.914332 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:05.999516 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.054421 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.102541 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.148902 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:06.183918 32047 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:06.184051 32047 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:06.184506 32047 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:06.185146 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:06.185837 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:06.185915 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:06.185987 31423 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:06.186137 32047 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:06.188180 32047 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:06.188294 32047 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:06.188730 32047 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
I20240501 14:09:06.189494 32047 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:06.189584 32047 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
W20240501 14:09:06.189682 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:06.189919 32047 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:06.190057 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:06.190109 31426 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:06.190249 32047 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:06.190754 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:06.191187 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:06.191257 31426 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:06.191376 32047 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:06.207682 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.304224 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.305035 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.467296 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.472610 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.585692 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:06.589530 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:06.640204 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.640211 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.761569 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.822413 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.822413 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:06.914970 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:07.015014 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.015014 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.081256 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.216754 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.216755 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.353650 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.423488 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.504305 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:07.568157 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:07.642879 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.720160 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:07.787248 32052 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:07.787389 32052 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:07.787874 32052 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:07.789005 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:07.789100 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:07.789157 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:07.789296 32052 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:07.805581 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.870388 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:07.915836 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:07.958369 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.101881 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.111196 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:08.177976 32052 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:08.178143 32052 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:08.178642 32052 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:08.179831 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:08.179961 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:08.180044 31423 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:08.180195 32052 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:08.192543 32052 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:08.192701 32052 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:08.193202 32052 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:08.194036 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:08.194255 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:08.194336 31423 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:08.194473 32052 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:08.269299 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.358150 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.446480 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:08.584266 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:08.603324 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.609472 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.780418 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.873207 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:08.925287 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:08.966781 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.143646 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.144456 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.425725 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.426429 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:09.581840 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:09.620033 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.715615 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:09.716455 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:09.874454 32056 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:09.874593 32056 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:09.874998 32056 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:09.876366 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:09.876530 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:09.876606 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:09.876747 32056 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:09.918221 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:09.966480 32056 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:09.966626 32056 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:09.967069 32056 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:09.968137 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:09.968222 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:09.968272 31426 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:09.968398 32056 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:09.979024 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:10.019239 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.020182 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.220007 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.328716 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.329743 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:10.406268 32056 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:10.406417 32056 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:10.406863 32056 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:10.407908 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:10.408054 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:10.408125 31423 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:10.408352 32056 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:09:10.535825 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:10.578333 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:10.647565 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.647586 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.860108 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:10.977576 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:11.029637 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:11.089433 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:11.318377 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:11.418572 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:11.541652 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:11.564319 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:11.664745 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:11.884842 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.011885 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.018090 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.079380 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:12.101316 32060 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:12.101447 32060 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:12.101892 32060 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:12.102494 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 14:09:12.103245 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:12.103325 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:12.103379 31423 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:12.103511 32060 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:12.140743 32060 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:12.140867 32060 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:12.141307 32060 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:12.142272 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:12.142362 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:12.142436 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:12.142576 32060 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:12.252478 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.379945 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:12.574937 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:12.617013 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.745916 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:12.751135 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:12.907271 32064 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:12.907403 32064 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:12.907842 32064 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:12.909107 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:12.909204 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:12.909261 31423 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:12.909379 32064 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:13.000559 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:13.119277 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:13.131906 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:13.381305 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:13.517150 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:13.521432 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:13.581034 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:13.779623 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:13.918408 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:14.123107 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:14.181833 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:14.181833 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:14.324743 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:14.567368 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:14.592754 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:14.738313 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:14.740224 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:15.020753 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:15.159570 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:15.164826 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:15.448061 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:15.563189 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:15.596618 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:15.598850 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:15.886220 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.042253 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.186460 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:16.193435 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.490532 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.493791 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:16.563768 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:16.799306 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.800132 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:16.958052 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:17.213768 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:17.265154 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:17.268292 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:17.427420 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:17.572208 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:17.741315 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:17.909309 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:17.909309 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:18.102802 32068 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:18.102933 32068 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:18.103375 32068 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:18.104022 31423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 14:09:18.104758 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:18.104840 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:18.104895 31426 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:18.105043 32068 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:09:18.225387 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:18.231472 31804 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:18.276875 32068 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:18.277048 32068 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:18.277568 32068 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:18.278604 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:18.278769 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:18.278851 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:18.279009 32068 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:18.395869 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:18.395905 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:18.582839 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:18.725139 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:18.893142 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:18.893142 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:19.230890 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:19.270331 32072 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:19.270462 32072 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:19.270887 32072 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:19.272110 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:19.272192 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:19.272240 31426 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:19.272383 32072 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:19.326383 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:19.399063 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:19.401011 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:19.581571 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:19.914036 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:19.916869 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:20.097095 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:20.365985 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:20.437469 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:20.561743 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:20.617137 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:20.806145 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:20.969348 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:21.170143 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:21.439791 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:21.514297 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:21.568210 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:21.704267 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:21.899168 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:22.067258 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:22.071162 32077 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:22.071278 32077 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:22.071677 32077 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:22.072993 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:22.073143 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:22.073220 31426 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:22.073387 32077 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:22.273010 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:22.440920 31804 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:22.576836 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:22.629671 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:22.820287 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:23.030419 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:23.196034 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:23.413581 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:23.503916 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:23.567852 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:23.775086 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:23.967900 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:24.175212 32081 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:24.175338 32081 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:24.175784 32081 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:24.176393 31423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 14:09:24.177080 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:24.177207 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:24.177289 31426 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:24.177429 32081 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:24.364385 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:24.386863 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:24.557834 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:24.585147 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:24.758579 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:24.963912 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:25.189122 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:25.495611 32085 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:25.495740 32085 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:25.496217 32085 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:25.497406 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:25.497538 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:25.497586 31423 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:25.497712 32085 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:25.568488 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:25.573540 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:25.577152 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:25.593693 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:25.977509 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:26.099906 32089 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:26.100033 32089 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:26.100452 32089 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:26.101642 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:26.101718 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:26.101764 31426 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:26.101902 32089 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:09:26.193746 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:26.416083 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:26.580008 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:26.596998 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:26.805274 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:26.817656 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:27.229666 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:27.449352 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:27.558893 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:27.650882 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:27.674376 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:28.079696 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:28.093894 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:28.513526 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:28.568805 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:28.731417 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:28.743541 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:28.970476 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:29.389770 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:29.403133 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:29.585501 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:29.741225 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:29.835876 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:30.077294 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:30.077294 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:30.509202 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:30.561095 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:30.754972 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:30.755867 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:30.983856 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:31.422668 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:31.440111 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:31.563032 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:31.794346 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:31.884153 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:32.130385 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:32.131307 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:32.562078 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:32.600241 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:32.826967 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:32.835352 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:33.301645 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:33.548890 31449 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:33.548921 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:33.566766 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:33.916059 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:34.268594 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:34.271780 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:34.568511 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:34.752287 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:34.815323 32094 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:34.815454 32094 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:34.815917 32094 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:34.816565 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 14:09:34.817268 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:34.817368 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:34.817422 31426 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:34.817543 32094 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:34.997666 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:35.000914 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:35.253683 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:35.381827 32098 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:35.381960 32098 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:35.382648 32098 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:35.384263 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:35.384351 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:35.384402 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:35.384608 32098 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:35.620901 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:35.739284 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:35.742198 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:35.996096 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:36.130501 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:36.490794 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:36.491693 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:36.560168 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:36.997740 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:36.998579 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:37.138090 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:37.222882 32102 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:37.223011 32102 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:37.223450 32102 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:37.224714 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:37.224794 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:37.224851 31426 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:37.224983 32102 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:37.256114 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:37.636399 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:37.757558 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:38.018610 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:38.024894 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:38.159518 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:38.533958 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:38.572062 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:38.806552 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:39.201913 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:39.309737 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:39.586102 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:39.588781 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:39.599401 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:40.114780 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:40.256199 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:40.396915 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:40.563189 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:40.587163 32106 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:40.587282 32106 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:40.587715 32106 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:40.588268 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 14:09:40.588907 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:40.589143 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:40.589217 31426 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:40.589349 32106 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:09:40.895715 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:41.184244 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:41.199584 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:41.327971 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:41.563076 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:41.728853 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:42.016275 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:42.415469 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:42.528326 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:42.568714 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:42.823689 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:42.840059 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:43.374655 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:43.521471 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:43.563342 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:43.672142 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:44.192019 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:44.488260 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:44.515839 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:44.566392 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:44.641024 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:45.050242 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:45.365046 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:45.561014 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:45.647986 31801 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:45.774408 32110 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:45.774549 32110 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:45.774956 32110 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:45.775527 31423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:09:45.776254 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:45.776363 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:45.776418 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:45.776546 32110 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:45.892040 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:46.195451 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:46.219732 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:46.572186 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:46.754112 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:46.773047 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:47.088790 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:47.563176 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:47.629532 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:47.802691 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:47.936823 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:47.964722 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:48.530560 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:48.564405 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:48.803282 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:48.851882 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:49.397029 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:49.547302 32114 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:49.547433 32114 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:49.547909 32114 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:49.549202 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:49.549357 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:49.549444 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:49.549598 32114 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:49.590875 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:49.724469 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:49.745735 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:49.878479 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:50.322899 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:50.565785 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:50.650454 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:50.895025 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:51.198326 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:51.534581 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:51.563258 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:51.565203 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:52.006073 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:52.148841 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:52.482393 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:52.567353 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:09:53.015275 32118 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:53.015417 32118 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:53.015902 32118 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:53.016548 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:09:53.017249 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:53.017323 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:53.017366 31426 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:53.017521 32118 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:53.041601 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:53.075985 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:53.387432 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:53.411842 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:53.581545 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:54.017179 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:54.177871 31804 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:54.346737 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:54.570446 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:54.919822 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:55.234854 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:55.281445 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:55.296893 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:55.568873 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:55.917258 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:56.236400 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:09:56.251648 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:56.559885 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:56.832476 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:57.203167 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:57.220623 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:57.369104 31801 meta_cache.cc:302] tablet 61752feefd0e4bac8dd22753f55ff1cc: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:57.389763 32122 raft_consensus.cc:482] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:57.389897 32122 raft_consensus.cc:504] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:57.390338 32122 leader_election.cc:288] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:57.391559 31426 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:57.391638 31423 leader_election.cc:334] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:57.391696 31423 leader_election.cc:302] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:57.391863 32122 raft_consensus.cc:2726] T b66a5f961f61423fa452a4491bbabc67 P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:57.570573 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:57.849766 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:58.192616 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:58.460958 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:58.514129 32126 raft_consensus.cc:482] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:58.514257 32126 raft_consensus.cc:504] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:58.514708 32126 leader_election.cc:288] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:58.515329 31426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:09:58.516100 31426 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:58.516180 31423 leader_election.cc:334] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:58.516227 31423 leader_election.cc:302] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:58.516367 32126 raft_consensus.cc:2726] T 61752feefd0e4bac8dd22753f55ff1cc P 954927af57d74f00b3868b538afc045a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:58.575742 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:58.787134 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:59.156519 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:59.174153 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:09:59.488993 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:09:59.586776 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:09:59.819859 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:09:59.952750 32130 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:09:59.952877 32130 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:09:59.953306 32130 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:09:59.956087 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
W20240501 14:09:59.956794 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:09:59.956888 31423 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:09:59.957070 32130 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:00.169473 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:00.502674 31801 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:10:00.587221 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:10:00.763666 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:01.152266 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:01.173812 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:01.511479 31801 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:10:01.563519 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:10:01.825402 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:02.188248 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:02.526430 31801 meta_cache.cc:302] tablet b66a5f961f61423fa452a4491bbabc67: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
I20240501 14:10:02.560298 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:10:02.788609 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:03.188262 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:03.212790 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:10:03.496176 32133 raft_consensus.cc:482] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a46c97abd14ff68d8d8465313e9d79)
I20240501 14:10:03.496316 32133 raft_consensus.cc:504] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954927af57d74f00b3868b538afc045a" member_type: VOTER last_known_addr { host: "127.30.146.65" port: 38615 } } peers { permanent_uuid: "36a46c97abd14ff68d8d8465313e9d79" member_type: VOTER last_known_addr { host: "127.30.146.66" port: 43151 } } peers { permanent_uuid: "aa5fb08274634c9198b5ca985e3239b7" member_type: VOTER last_known_addr { host: "127.30.146.67" port: 39395 } }
I20240501 14:10:03.496747 32133 leader_election.cc:288] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151), aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395)
W20240501 14:10:03.498037 31423 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395): Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:10:03.498127 31426 leader_election.cc:334] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151): Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:10:03.498188 31426 leader_election.cc:302] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954927af57d74f00b3868b538afc045a; no voters: 36a46c97abd14ff68d8d8465313e9d79, aa5fb08274634c9198b5ca985e3239b7
I20240501 14:10:03.498342 32133 raft_consensus.cc:2726] T 462974e1e7cc439b87d95a99c93258bf P 954927af57d74f00b3868b538afc045a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:03.530292 31804 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica 36a46c97abd14ff68d8d8465313e9d79 (127.30.146.66:43151) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.66:43151: connect: Connection refused (error 111)
I20240501 14:10:03.573490 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:10:03.875332 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
W20240501 14:10:04.243111 31450 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37260: Illegal state: replica 954927af57d74f00b3868b538afc045a is not leader of this config: current role FOLLOWER
I20240501 14:10:04.566105 31876 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 14:10:04.566280 31801 meta_cache.cc:302] tablet 462974e1e7cc439b87d95a99c93258bf: replica aa5fb08274634c9198b5ca985e3239b7 (127.30.146.67:39395) has failed: Network error: Client connection negotiation failed: client connection to 127.30.146.67:39395: connect: Connection refused (error 111)
W20240501 14:10:04.850914 31804 batcher.cc:441] Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:04.851287 31305 client-test-util.cc:49] 5 failed ops. First 10 errors follow
I20240501 14:10:04.851367 31305 client-test-util.cc:55] Op INSERT int64 rand_key=3186124404309622817, int64 link_to=-163136436715782111, int64 insert_ts=1714572544682388 had status Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:04.851523 31305 client-test-util.cc:55] Op INSERT int64 rand_key=3369586751144919077, int64 link_to=-2637678724480171995, int64 insert_ts=1714572544682535 had status Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:04.851555 31305 client-test-util.cc:55] Op INSERT int64 rand_key=3553189835468505129, int64 link_to=-5112361749732982743, int64 insert_ts=1714572544682666 had status Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:04.851593 31305 client-test-util.cc:55] Op INSERT int64 rand_key=3736933657280512045, int64 link_to=-7586763300009082835, int64 insert_ts=1714572544682786 had status Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:04.851613 31305 client-test-util.cc:55] Op INSERT int64 rand_key=3920396004115808305, int64 link_to=8385157010959368241, int64 insert_ts=1714572544682905 had status Timed out: Failed to write batch of 5 ops to tablet 462974e1e7cc439b87d95a99c93258bf after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 462974e1e7cc439b87d95a99c93258bf, num_ops: 5, num_attempts: 343) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
F20240501 14:10:04.851656 31305 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714572604 (unix time) try "date -d @1714572604" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007a49) received by PID 31305 (TID 0x7fbf0c867280) from PID 31305; stack trace: ***
@ 0x7fbf18b7e980 (unknown) at ??:0
@ 0x7fbf16802fb7 gsignal at ??:0
@ 0x7fbf16804921 abort at ??:0
@ 0x7fbf1770fdcd google::LogMessage::Fail() at ??:0
@ 0x7fbf17713b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbf1770f7cc google::LogMessage::Flush() at ??:0
@ 0x7fbf17710f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf19e417f2 kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x55ba624ec236 kudu::client::FlushSessionOrDie() at ??:0
@ 0x55ba624efcaa kudu::LinkedListTester::LoadLinkedList() at ??:0
@ 0x55ba624e54ca kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at ??:0
@ 0x7fbf186060ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fbf185fabdb testing::Test::Run() at ??:0
@ 0x7fbf185fad9d testing::TestInfo::Run() at ??:0
@ 0x7fbf185fb377 testing::TestSuite::Run() at ??:0
@ 0x7fbf185fb77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fbf1860660d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fbf185fae63 testing::UnitTest::Run() at ??:0
@ 0x7fbf19e531de RUN_ALL_TESTS() at ??:0
@ 0x7fbf19e5253d main at ??:0
@ 0x7fbf167e5bf7 __libc_start_main at ??:0
@ 0x55ba624e334a _start at ??:0