[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 10:00:32.154173 13465 test_util.cc:268] Using random seed: -110646769
I20240426 10:00:32.166754 13465 linked_list-test.cc:100] Linked List Test Configuration:
I20240426 10:00:32.166847 13465 linked_list-test.cc:101] --------------
I20240426 10:00:32.166889 13465 linked_list-test.cc:102] 50 chains
I20240426 10:00:32.166937 13465 linked_list-test.cc:103] 3 tablets
I20240426 10:00:32.167003 13465 linked_list-test.cc:104] Mutations on
I20240426 10:00:32.167055 13465 linked_list-test.cc:105] --------------
I20240426 10:00:32.167321 13465 ts_itest-base.cc:112] Starting cluster with:
I20240426 10:00:32.167379 13465 ts_itest-base.cc:113] --------------
I20240426 10:00:32.167443 13465 ts_itest-base.cc:114] 3 tablet servers
I20240426 10:00:32.167493 13465 ts_itest-base.cc:115] 3 replicas per TS
I20240426 10:00:32.167554 13465 ts_itest-base.cc:116] --------------
2024-04-26T10:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-26T10:00:32Z Disabled control of system clock
I20240426 10:00:32.191849 13465 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBCpadc/build/asan/bin/kudu
/tmp/dist-test-taskBCpadc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.38.126:35363
--webserver_interface=127.13.38.126
--webserver_port=0
--builtin_ntp_servers=127.13.38.84:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.38.126:35363
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240426 10:00:32.519488 13477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 10:00:32.519815 13477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 10:00:32.519927 13477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 10:00:32.527499 13477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240426 10:00:32.527624 13477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 10:00:32.527676 13477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240426 10:00:32.527734 13477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240426 10:00:32.537503 13477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.38.84:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.38.126:35363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.38.126:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.13.38.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 10:00:32.538902 13477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 10:00:32.540606 13477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 10:00:32.552626 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:32.553462 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:32.554003 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 10:00:32.554615 13477 server_base.cc:1034] running on GCE node
I20240426 10:00:32.555456 13477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 10:00:32.556990 13477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 10:00:32.558210 13477 hybrid_clock.cc:648] HybridClock initialized: now 1714125632558132 us; error 62 us; skew 500 ppm
I20240426 10:00:32.558547 13477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 10:00:32.572629 13477 webserver.cc:460] Webserver started at http://127.13.38.126:37637/ using document root <none> and password file <none>
I20240426 10:00:32.573470 13477 fs_manager.cc:359] Metadata directory not provided
I20240426 10:00:32.573616 13477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 10:00:32.574002 13477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 10:00:32.576843 13477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data/instance:
uuid: "df22014c12274aa3a526325c05c46a07"
format_stamp: "Formatted at 2024-04-26 10:00:32 on dist-test-slave-8csj"
I20240426 10:00:32.577641 13477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal/instance:
uuid: "df22014c12274aa3a526325c05c46a07"
format_stamp: "Formatted at 2024-04-26 10:00:32 on dist-test-slave-8csj"
I20240426 10:00:32.583199 13477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240426 10:00:32.586918 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:32.588416 13477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240426 10:00:32.588598 13477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal
uuid: "df22014c12274aa3a526325c05c46a07"
format_stamp: "Formatted at 2024-04-26 10:00:32 on dist-test-slave-8csj"
I20240426 10:00:32.588793 13477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 10:00:32.603797 13477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 10:00:32.604707 13477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 10:00:32.605022 13477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 10:00:32.624470 13477 rpc_server.cc:307] RPC server started. Bound to: 127.13.38.126:35363
I20240426 10:00:32.624490 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.38.126:35363 every 8 connection(s)
I20240426 10:00:32.625986 13477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/data/info.pb
I20240426 10:00:32.629343 13465 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBCpadc/build/asan/bin/kudu as pid 13477
I20240426 10:00:32.629578 13465 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/master-0/wal/instance
I20240426 10:00:32.629899 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:32.641062 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07: Bootstrap starting.
I20240426 10:00:32.644167 13543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:32.645352 13543 log.cc:824] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07: Log is configured to *not* fsync() on all Append() calls
I20240426 10:00:32.648036 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07: No bootstrap required, opened a new log
I20240426 10:00:32.656752 13543 raft_consensus.cc:348] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } }
I20240426 10:00:32.657155 13543 raft_consensus.cc:374] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:32.657259 13543 raft_consensus.cc:729] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df22014c12274aa3a526325c05c46a07, State: Initialized, Role: FOLLOWER
I20240426 10:00:32.657915 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } }
I20240426 10:00:32.658141 13543 raft_consensus.cc:388] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 10:00:32.658285 13543 raft_consensus.cc:482] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 10:00:32.658435 13543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:32.660517 13543 raft_consensus.cc:504] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } }
I20240426 10:00:32.661002 13543 leader_election.cc:302] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df22014c12274aa3a526325c05c46a07; no voters:
I20240426 10:00:32.661621 13543 leader_election.cc:288] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 10:00:32.661756 13548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 10:00:32.662489 13548 raft_consensus.cc:686] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [term 1 LEADER]: Becoming Leader. State: Replica: df22014c12274aa3a526325c05c46a07, State: Running, Role: LEADER
I20240426 10:00:32.663041 13548 consensus_queue.cc:237] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } }
I20240426 10:00:32.663718 13543 sys_catalog.cc:564] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 10:00:32.666451 13549 sys_catalog.cc:455] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df22014c12274aa3a526325c05c46a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } } }
I20240426 10:00:32.666990 13549 sys_catalog.cc:458] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [sys.catalog]: This master's current role is: LEADER
I20240426 10:00:32.667032 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df22014c12274aa3a526325c05c46a07. Latest consensus state: current_term: 1 leader_uuid: "df22014c12274aa3a526325c05c46a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df22014c12274aa3a526325c05c46a07" member_type: VOTER last_known_addr { host: "127.13.38.126" port: 35363 } } }
I20240426 10:00:32.667310 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07 [sys.catalog]: This master's current role is: LEADER
I20240426 10:00:32.668579 13556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 10:00:32.674867 13556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 10:00:32.685256 13556 catalog_manager.cc:1347] Generated new cluster ID: bd426fe908004199b4e9d2596c7610db
I20240426 10:00:32.685395 13556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 10:00:32.710281 13556 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 10:00:32.712050 13556 catalog_manager.cc:1504] Loading token signing keys...
I20240426 10:00:32.725525 13556 catalog_manager.cc:5857] T 00000000000000000000000000000000 P df22014c12274aa3a526325c05c46a07: Generated new TSK 0
I20240426 10:00:32.726480 13556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 10:00:32.734799 13465 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBCpadc/build/asan/bin/kudu
/tmp/dist-test-taskBCpadc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.38.65:0
--local_ip_for_outbound_sockets=127.13.38.65
--webserver_interface=127.13.38.65
--webserver_port=0
--tserver_master_addrs=127.13.38.126:35363
--builtin_ntp_servers=127.13.38.84:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240426 10:00:33.110018 13567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 10:00:33.110332 13567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 10:00:33.110491 13567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 10:00:33.118572 13567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 10:00:33.118739 13567 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 10:00:33.118810 13567 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 10:00:33.118893 13567 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 10:00:33.118999 13567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.38.65
I20240426 10:00:33.128433 13567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.38.84:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.38.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.13.38.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.13.38.126:35363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.38.65
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 10:00:33.129803 13567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 10:00:33.131634 13567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 10:00:33.144789 13572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:33.145440 13575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:33.146294 13573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 10:00:33.147032 13567 server_base.cc:1034] running on GCE node
I20240426 10:00:33.147814 13567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 10:00:33.148846 13567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 10:00:33.150068 13567 hybrid_clock.cc:648] HybridClock initialized: now 1714125633150015 us; error 41 us; skew 500 ppm
I20240426 10:00:33.150421 13567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 10:00:33.153343 13567 webserver.cc:460] Webserver started at http://127.13.38.65:45799/ using document root <none> and password file <none>
I20240426 10:00:33.154151 13567 fs_manager.cc:359] Metadata directory not provided
I20240426 10:00:33.154268 13567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 10:00:33.154592 13567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 10:00:33.157356 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data/instance:
uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.158134 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal/instance:
uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.163385 13567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240426 10:00:33.167073 13581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.168571 13567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240426 10:00:33.168774 13567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal
uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.169016 13567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 10:00:33.187518 13567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 10:00:33.188376 13567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 10:00:33.188639 13567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 10:00:33.189548 13567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 10:00:33.191012 13567 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 10:00:33.191102 13567 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.191181 13567 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 10:00:33.191236 13567 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.226290 13567 rpc_server.cc:307] RPC server started. Bound to: 127.13.38.65:44945
I20240426 10:00:33.226343 13693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.38.65:44945 every 8 connection(s)
I20240426 10:00:33.227607 13567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/data/info.pb
I20240426 10:00:33.235064 13465 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBCpadc/build/asan/bin/kudu as pid 13567
I20240426 10:00:33.235299 13465 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-0/wal/instance
I20240426 10:00:33.238580 13465 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBCpadc/build/asan/bin/kudu
/tmp/dist-test-taskBCpadc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.38.66:0
--local_ip_for_outbound_sockets=127.13.38.66
--webserver_interface=127.13.38.66
--webserver_port=0
--tserver_master_addrs=127.13.38.126:35363
--builtin_ntp_servers=127.13.38.84:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 10:00:33.241393 13694 heartbeater.cc:347] Connected to a master server at 127.13.38.126:35363
I20240426 10:00:33.241699 13694 heartbeater.cc:464] Registering TS with master...
I20240426 10:00:33.242601 13694 heartbeater.cc:511] Master 127.13.38.126:35363 requested a full tablet report, sending...
I20240426 10:00:33.244933 13508 ts_manager.cc:194] Registered new tserver with Master: 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945)
I20240426 10:00:33.247243 13508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.38.65:45541
W20240426 10:00:33.589267 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 10:00:33.589577 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 10:00:33.589695 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 10:00:33.597334 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 10:00:33.597491 13698 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 10:00:33.597527 13698 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 10:00:33.597581 13698 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 10:00:33.597662 13698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.38.66
I20240426 10:00:33.607362 13698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.38.84:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.38.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.13.38.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.13.38.126:35363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.38.66
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 10:00:33.608708 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 10:00:33.610505 13698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 10:00:33.623405 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:33.624064 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:33.624960 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 10:00:33.625887 13698 server_base.cc:1034] running on GCE node
I20240426 10:00:33.626619 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 10:00:33.627826 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 10:00:33.629066 13698 hybrid_clock.cc:648] HybridClock initialized: now 1714125633629005 us; error 59 us; skew 500 ppm
I20240426 10:00:33.629462 13698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 10:00:33.632370 13698 webserver.cc:460] Webserver started at http://127.13.38.66:44413/ using document root <none> and password file <none>
I20240426 10:00:33.633361 13698 fs_manager.cc:359] Metadata directory not provided
I20240426 10:00:33.633483 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 10:00:33.633970 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 10:00:33.638191 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data/instance:
uuid: "66c3b02bff40431aaed66dba0405aed7"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.639200 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal/instance:
uuid: "66c3b02bff40431aaed66dba0405aed7"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.646162 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240426 10:00:33.649734 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.651309 13698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240426 10:00:33.651495 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal
uuid: "66c3b02bff40431aaed66dba0405aed7"
format_stamp: "Formatted at 2024-04-26 10:00:33 on dist-test-slave-8csj"
I20240426 10:00:33.651723 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 10:00:33.666951 13698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 10:00:33.667829 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 10:00:33.668143 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 10:00:33.669131 13698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 10:00:33.670758 13698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 10:00:33.670861 13698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.670980 13698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 10:00:33.671056 13698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:33.706231 13698 rpc_server.cc:307] RPC server started. Bound to: 127.13.38.66:46215
I20240426 10:00:33.706292 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.38.66:46215 every 8 connection(s)
I20240426 10:00:33.707599 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/data/info.pb
I20240426 10:00:33.717739 13465 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBCpadc/build/asan/bin/kudu as pid 13698
I20240426 10:00:33.717964 13465 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-1/wal/instance
I20240426 10:00:33.721627 13465 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBCpadc/build/asan/bin/kudu
/tmp/dist-test-taskBCpadc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.38.67:0
--local_ip_for_outbound_sockets=127.13.38.67
--webserver_interface=127.13.38.67
--webserver_port=0
--tserver_master_addrs=127.13.38.126:35363
--builtin_ntp_servers=127.13.38.84:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 10:00:33.722864 13825 heartbeater.cc:347] Connected to a master server at 127.13.38.126:35363
I20240426 10:00:33.723237 13825 heartbeater.cc:464] Registering TS with master...
I20240426 10:00:33.724341 13825 heartbeater.cc:511] Master 127.13.38.126:35363 requested a full tablet report, sending...
I20240426 10:00:33.726070 13508 ts_manager.cc:194] Registered new tserver with Master: 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
I20240426 10:00:33.727083 13508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.38.66:51497
W20240426 10:00:34.075381 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 10:00:34.075696 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 10:00:34.075816 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 10:00:34.084056 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 10:00:34.084234 13828 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 10:00:34.084273 13828 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 10:00:34.084332 13828 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 10:00:34.084420 13828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.38.67
I20240426 10:00:34.094688 13828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.38.84:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.38.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.13.38.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.13.38.126:35363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.38.67
--log_dir=/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 10:00:34.096073 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 10:00:34.097857 13828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 10:00:34.111348 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:34.111914 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 10:00:34.112721 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 10:00:34.113653 13828 server_base.cc:1034] running on GCE node
I20240426 10:00:34.114347 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 10:00:34.115366 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 10:00:34.116559 13828 hybrid_clock.cc:648] HybridClock initialized: now 1714125634116506 us; error 56 us; skew 500 ppm
I20240426 10:00:34.116868 13828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 10:00:34.119311 13828 webserver.cc:460] Webserver started at http://127.13.38.67:38021/ using document root <none> and password file <none>
I20240426 10:00:34.120131 13828 fs_manager.cc:359] Metadata directory not provided
I20240426 10:00:34.120234 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 10:00:34.120565 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 10:00:34.123433 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data/instance:
uuid: "755ae40e4b834d9e8bdc3ae0cc726af2"
format_stamp: "Formatted at 2024-04-26 10:00:34 on dist-test-slave-8csj"
I20240426 10:00:34.124229 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal/instance:
uuid: "755ae40e4b834d9e8bdc3ae0cc726af2"
format_stamp: "Formatted at 2024-04-26 10:00:34 on dist-test-slave-8csj"
I20240426 10:00:34.130309 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240426 10:00:34.133893 13842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:34.135277 13828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240426 10:00:34.135466 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal
uuid: "755ae40e4b834d9e8bdc3ae0cc726af2"
format_stamp: "Formatted at 2024-04-26 10:00:34 on dist-test-slave-8csj"
I20240426 10:00:34.135679 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 10:00:34.172108 13828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 10:00:34.172981 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 10:00:34.173302 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 10:00:34.174376 13828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 10:00:34.175913 13828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 10:00:34.176002 13828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:34.176093 13828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 10:00:34.176147 13828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 10:00:34.212595 13828 rpc_server.cc:307] RPC server started. Bound to: 127.13.38.67:37307
I20240426 10:00:34.212641 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.38.67:37307 every 8 connection(s)
I20240426 10:00:34.213970 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/data/info.pb
I20240426 10:00:34.226923 13465 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBCpadc/build/asan/bin/kudu as pid 13828
I20240426 10:00:34.227157 13465 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBCpadc/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125632081207-13465-0/linked-list-cluster/ts-2/wal/instance
I20240426 10:00:34.233351 13955 heartbeater.cc:347] Connected to a master server at 127.13.38.126:35363
I20240426 10:00:34.233700 13955 heartbeater.cc:464] Registering TS with master...
I20240426 10:00:34.235411 13955 heartbeater.cc:511] Master 127.13.38.126:35363 requested a full tablet report, sending...
I20240426 10:00:34.237226 13508 ts_manager.cc:194] Registered new tserver with Master: 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
I20240426 10:00:34.238493 13508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.38.67:56183
I20240426 10:00:34.249761 13465 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 10:00:34.250427 13694 heartbeater.cc:503] Master 127.13.38.126:35363 was elected leader, sending a full tablet report...
I20240426 10:00:34.273065 13508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56468:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240426 10:00:34.279690 13508 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 10:00:34.311311 13760 tablet_service.cc:1467] Processing CreateTablet for tablet be24d5b36bce438abca45fe308c21b73 (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 10:00:34.310495 13889 tablet_service.cc:1467] Processing CreateTablet for tablet 1c421916024e46a8bbe38590663e721e (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 10:00:34.312510 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be24d5b36bce438abca45fe308c21b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.312516 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c421916024e46a8bbe38590663e721e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.313954 13628 tablet_service.cc:1467] Processing CreateTablet for tablet 1c421916024e46a8bbe38590663e721e (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 10:00:34.315935 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c421916024e46a8bbe38590663e721e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.310436 13759 tablet_service.cc:1467] Processing CreateTablet for tablet 1c421916024e46a8bbe38590663e721e (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 10:00:34.318296 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c421916024e46a8bbe38590663e721e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.318354 13888 tablet_service.cc:1467] Processing CreateTablet for tablet a0e82ff0b52944bea2d643458a02405a (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 10:00:34.318876 13888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e82ff0b52944bea2d643458a02405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.320034 13758 tablet_service.cc:1467] Processing CreateTablet for tablet a0e82ff0b52944bea2d643458a02405a (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 10:00:34.320649 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e82ff0b52944bea2d643458a02405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.322131 13627 tablet_service.cc:1467] Processing CreateTablet for tablet a0e82ff0b52944bea2d643458a02405a (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 10:00:34.322662 13627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e82ff0b52944bea2d643458a02405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.324446 13890 tablet_service.cc:1467] Processing CreateTablet for tablet be24d5b36bce438abca45fe308c21b73 (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 10:00:34.325039 13890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be24d5b36bce438abca45fe308c21b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.330314 13629 tablet_service.cc:1467] Processing CreateTablet for tablet be24d5b36bce438abca45fe308c21b73 (DEFAULT_TABLE table=TestTable [id=afa829b0479b4498bd139ab0fae734dd]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 10:00:34.330936 13629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be24d5b36bce438abca45fe308c21b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 10:00:34.334787 13973 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: Bootstrap starting.
I20240426 10:00:34.340274 13974 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: Bootstrap starting.
I20240426 10:00:34.343402 13973 tablet_bootstrap.cc:654] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.344962 13973 log.cc:824] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: Log is configured to *not* fsync() on all Append() calls
I20240426 10:00:34.345489 13974 tablet_bootstrap.cc:654] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.347133 13974 log.cc:824] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: Log is configured to *not* fsync() on all Append() calls
I20240426 10:00:34.349565 13975 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: Bootstrap starting.
I20240426 10:00:34.353519 13973 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: No bootstrap required, opened a new log
I20240426 10:00:34.353818 13973 ts_tablet_manager.cc:1367] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20240426 10:00:34.353966 13974 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: No bootstrap required, opened a new log
I20240426 10:00:34.354259 13974 ts_tablet_manager.cc:1367] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240426 10:00:34.354517 13975 tablet_bootstrap.cc:654] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.355926 13975 log.cc:824] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: Log is configured to *not* fsync() on all Append() calls
I20240426 10:00:34.359401 13975 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: No bootstrap required, opened a new log
I20240426 10:00:34.359741 13975 ts_tablet_manager.cc:1367] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240426 10:00:34.361491 13973 raft_consensus.cc:348] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.361954 13973 raft_consensus.cc:374] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.362077 13973 raft_consensus.cc:729] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755ae40e4b834d9e8bdc3ae0cc726af2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.362761 13973 consensus_queue.cc:260] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.363723 13974 raft_consensus.cc:348] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.364235 13974 raft_consensus.cc:374] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.364382 13974 raft_consensus.cc:729] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c3b02bff40431aaed66dba0405aed7, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.364675 13973 ts_tablet_manager.cc:1398] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20240426 10:00:34.365279 13955 heartbeater.cc:503] Master 127.13.38.126:35363 was elected leader, sending a full tablet report...
I20240426 10:00:34.365204 13974 consensus_queue.cc:260] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.366984 13825 heartbeater.cc:503] Master 127.13.38.126:35363 was elected leader, sending a full tablet report...
I20240426 10:00:34.367188 13973 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2: Bootstrap starting.
I20240426 10:00:34.367640 13974 ts_tablet_manager.cc:1398] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7: Time spent starting tablet: real 0.013s user 0.002s sys 0.010s
I20240426 10:00:34.368083 13974 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: Bootstrap starting.
I20240426 10:00:34.368999 13973 tablet_bootstrap.cc:654] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.369868 13974 tablet_bootstrap.cc:654] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.371698 13975 raft_consensus.cc:348] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.372615 13975 raft_consensus.cc:374] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.372825 13975 raft_consensus.cc:729] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2491b640d5ae43ba955f47ab77ef7bb2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.374105 13975 consensus_queue.cc:260] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.376801 13975 ts_tablet_manager.cc:1398] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20240426 10:00:34.377465 13975 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: Bootstrap starting.
I20240426 10:00:34.378144 13974 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: No bootstrap required, opened a new log
I20240426 10:00:34.378319 13974 ts_tablet_manager.cc:1367] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240426 10:00:34.379469 13975 tablet_bootstrap.cc:654] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.379343 13974 raft_consensus.cc:348] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.379583 13974 raft_consensus.cc:374] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.379679 13974 raft_consensus.cc:729] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c3b02bff40431aaed66dba0405aed7, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.379982 13974 consensus_queue.cc:260] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.381769 13973 tablet_bootstrap.cc:492] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2: No bootstrap required, opened a new log
I20240426 10:00:34.381801 13974 ts_tablet_manager.cc:1398] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240426 10:00:34.381970 13973 ts_tablet_manager.cc:1367] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240426 10:00:34.382161 13974 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7: Bootstrap starting.
I20240426 10:00:34.383004 13973 raft_consensus.cc:348] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.383239 13973 raft_consensus.cc:374] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.383358 13973 raft_consensus.cc:729] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755ae40e4b834d9e8bdc3ae0cc726af2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.383666 13973 consensus_queue.cc:260] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.384263 13973 ts_tablet_manager.cc:1398] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240426 10:00:34.384356 13974 tablet_bootstrap.cc:654] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.384542 13973 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: Bootstrap starting.
I20240426 10:00:34.390003 13973 tablet_bootstrap.cc:654] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.390601 13975 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: No bootstrap required, opened a new log
I20240426 10:00:34.390827 13975 ts_tablet_manager.cc:1367] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240426 10:00:34.391981 13973 tablet_bootstrap.cc:492] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: No bootstrap required, opened a new log
I20240426 10:00:34.391866 13975 raft_consensus.cc:348] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.392077 13975 raft_consensus.cc:374] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.392150 13973 ts_tablet_manager.cc:1367] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240426 10:00:34.392175 13975 raft_consensus.cc:729] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2491b640d5ae43ba955f47ab77ef7bb2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.392305 13974 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7: No bootstrap required, opened a new log
I20240426 10:00:34.392481 13974 ts_tablet_manager.cc:1367] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240426 10:00:34.392490 13975 consensus_queue.cc:260] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.392874 13973 raft_consensus.cc:348] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.393072 13975 ts_tablet_manager.cc:1398] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240426 10:00:34.393110 13973 raft_consensus.cc:374] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.393183 13973 raft_consensus.cc:729] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755ae40e4b834d9e8bdc3ae0cc726af2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.393189 13974 raft_consensus.cc:348] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.393452 13974 raft_consensus.cc:374] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.393527 13974 raft_consensus.cc:729] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c3b02bff40431aaed66dba0405aed7, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.393476 13973 consensus_queue.cc:260] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.393666 13975 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2: Bootstrap starting.
I20240426 10:00:34.393821 13974 consensus_queue.cc:260] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.394199 13973 ts_tablet_manager.cc:1398] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240426 10:00:34.395105 13974 ts_tablet_manager.cc:1398] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240426 10:00:34.395380 13975 tablet_bootstrap.cc:654] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2: Neither blocks nor log segments found. Creating new log.
I20240426 10:00:34.399250 13975 tablet_bootstrap.cc:492] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2: No bootstrap required, opened a new log
I20240426 10:00:34.399412 13975 ts_tablet_manager.cc:1367] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240426 10:00:34.400207 13975 raft_consensus.cc:348] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.400473 13975 raft_consensus.cc:374] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 10:00:34.400560 13975 raft_consensus.cc:729] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2491b640d5ae43ba955f47ab77ef7bb2, State: Initialized, Role: FOLLOWER
I20240426 10:00:34.400828 13975 consensus_queue.cc:260] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.401367 13975 ts_tablet_manager.cc:1398] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240426 10:00:34.407336 13980 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 10:00:34.407601 13980 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.409355 13980 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
I20240426 10:00:34.417021 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c421916024e46a8bbe38590663e721e" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" is_pre_election: true
I20240426 10:00:34.417024 13649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c421916024e46a8bbe38590663e721e" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" is_pre_election: true
I20240426 10:00:34.417539 13910 raft_consensus.cc:2454] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 0.
I20240426 10:00:34.417538 13649 raft_consensus.cc:2454] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 0.
I20240426 10:00:34.418171 13713 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 66c3b02bff40431aaed66dba0405aed7; no voters:
I20240426 10:00:34.418542 13980 raft_consensus.cc:2781] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 10:00:34.418668 13980 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 10:00:34.418771 13980 raft_consensus.cc:3037] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.421039 13980 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.421640 13980 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 1 election: Requested vote from peers 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
I20240426 10:00:34.422189 13649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c421916024e46a8bbe38590663e721e" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
I20240426 10:00:34.422259 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c421916024e46a8bbe38590663e721e" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2"
I20240426 10:00:34.422384 13649 raft_consensus.cc:3037] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.422472 13910 raft_consensus.cc:3037] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.424770 13649 raft_consensus.cc:2454] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 1.
I20240426 10:00:34.424762 13910 raft_consensus.cc:2454] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 1.
I20240426 10:00:34.425192 13713 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 66c3b02bff40431aaed66dba0405aed7; no voters:
I20240426 10:00:34.425527 13980 raft_consensus.cc:2781] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 10:00:34.426496 13980 raft_consensus.cc:686] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [term 1 LEADER]: Becoming Leader. State: Replica: 66c3b02bff40431aaed66dba0405aed7, State: Running, Role: LEADER
I20240426 10:00:34.426954 13980 consensus_queue.cc:237] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.431998 13508 catalog_manager.cc:5484] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66). New cstate: current_term: 1 leader_uuid: "66c3b02bff40431aaed66dba0405aed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } health_report { overall_health: UNKNOWN } } }
I20240426 10:00:34.456337 13979 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 10:00:34.456524 13979 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.458355 13979 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945)
W20240426 10:00:34.459728 13826 tablet.cc:2369] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240426 10:00:34.465962 13956 tablet.cc:2369] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 10:00:34.466454 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c3b02bff40431aaed66dba0405aed7" is_pre_election: true
I20240426 10:00:34.466845 13780 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755ae40e4b834d9e8bdc3ae0cc726af2 in term 0.
I20240426 10:00:34.466895 13649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" is_pre_election: true
I20240426 10:00:34.467162 13649 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755ae40e4b834d9e8bdc3ae0cc726af2 in term 0.
I20240426 10:00:34.467388 13845 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2; no voters:
I20240426 10:00:34.467844 13979 raft_consensus.cc:2781] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 10:00:34.467973 13979 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 10:00:34.468065 13979 raft_consensus.cc:3037] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.470096 13979 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:34.470786 13979 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [CANDIDATE]: Term 1 election: Requested vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945)
I20240426 10:00:34.471333 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c3b02bff40431aaed66dba0405aed7"
I20240426 10:00:34.471433 13649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
I20240426 10:00:34.471555 13780 raft_consensus.cc:3037] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.471652 13649 raft_consensus.cc:3037] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.474674 13780 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755ae40e4b834d9e8bdc3ae0cc726af2 in term 1.
I20240426 10:00:34.474661 13649 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755ae40e4b834d9e8bdc3ae0cc726af2 in term 1.
I20240426 10:00:34.475147 13845 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2; no voters:
I20240426 10:00:34.475515 13979 raft_consensus.cc:2781] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 10:00:34.476009 13979 raft_consensus.cc:686] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 LEADER]: Becoming Leader. State: Replica: 755ae40e4b834d9e8bdc3ae0cc726af2, State: Running, Role: LEADER
I20240426 10:00:34.476564 13979 consensus_queue.cc:237] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
W20240426 10:00:34.480078 13695 tablet.cc:2369] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 10:00:34.481462 13508 catalog_manager.cc:5484] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67). New cstate: current_term: 1 leader_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20240426 10:00:34.582165 13983 raft_consensus.cc:482] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 10:00:34.582512 13983 raft_consensus.cc:504] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.584229 13983 leader_election.cc:288] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
I20240426 10:00:34.591735 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e82ff0b52944bea2d643458a02405a" candidate_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c3b02bff40431aaed66dba0405aed7" is_pre_election: true
I20240426 10:00:34.591732 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e82ff0b52944bea2d643458a02405a" candidate_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" is_pre_election: true
I20240426 10:00:34.591991 13780 raft_consensus.cc:2454] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2491b640d5ae43ba955f47ab77ef7bb2 in term 0.
I20240426 10:00:34.591985 13910 raft_consensus.cc:2454] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2491b640d5ae43ba955f47ab77ef7bb2 in term 0.
I20240426 10:00:34.592473 13584 leader_election.cc:302] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 66c3b02bff40431aaed66dba0405aed7; no voters:
I20240426 10:00:34.592897 13983 raft_consensus.cc:2781] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 10:00:34.593034 13983 raft_consensus.cc:482] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 10:00:34.593138 13983 raft_consensus.cc:3037] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.595352 13983 raft_consensus.cc:504] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.596166 13983 leader_election.cc:288] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
I20240426 10:00:34.596563 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e82ff0b52944bea2d643458a02405a" candidate_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c3b02bff40431aaed66dba0405aed7"
I20240426 10:00:34.596674 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e82ff0b52944bea2d643458a02405a" candidate_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2"
I20240426 10:00:34.596793 13780 raft_consensus.cc:3037] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.596900 13910 raft_consensus.cc:3037] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 0 FOLLOWER]: Advancing to term 1
I20240426 10:00:34.598819 13780 raft_consensus.cc:2454] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2491b640d5ae43ba955f47ab77ef7bb2 in term 1.
I20240426 10:00:34.598816 13910 raft_consensus.cc:2454] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2491b640d5ae43ba955f47ab77ef7bb2 in term 1.
I20240426 10:00:34.599267 13582 leader_election.cc:302] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 755ae40e4b834d9e8bdc3ae0cc726af2; no voters:
I20240426 10:00:34.599768 13983 raft_consensus.cc:2781] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 10:00:34.600265 13983 raft_consensus.cc:686] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 LEADER]: Becoming Leader. State: Replica: 2491b640d5ae43ba955f47ab77ef7bb2, State: Running, Role: LEADER
I20240426 10:00:34.600801 13983 consensus_queue.cc:237] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:00:34.605355 13508 catalog_manager.cc:5484] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65). New cstate: current_term: 1 leader_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } health_report { overall_health: UNKNOWN } } }
I20240426 10:00:34.670007 13465 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 10:00:34.672950 13465 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 2491b640d5ae43ba955f47ab77ef7bb2 to finish bootstrapping
I20240426 10:00:34.683406 13465 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 66c3b02bff40431aaed66dba0405aed7 to finish bootstrapping
I20240426 10:00:34.691179 13465 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 755ae40e4b834d9e8bdc3ae0cc726af2 to finish bootstrapping
I20240426 10:00:34.699793 14000 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.13.38.66:44413/metrics
http://127.13.38.126:37637/threadz
http://127.13.38.126:37637/tablet-servers
http://127.13.38.67:38021/tablets
http://127.13.38.67:38021/mem-trackers
http://127.13.38.65:45799/mem-trackers
http://127.13.38.126:37637/metrics
http://127.13.38.67:38021/threadz
http://127.13.38.126:37637/ipki-ca-cert
http://127.13.38.66:44413/maintenance-manager
http://127.13.38.65:45799/maintenance-manager
http://127.13.38.65:45799/metrics
http://127.13.38.126:37637/dump-entities
http://127.13.38.66:44413/stacks
http://127.13.38.66:44413/threadz?group=all
http://127.13.38.65:45799/threadz
http://127.13.38.65:45799/transactions?tablet_id=a0e82ff0b52944bea2d643458a02405a
http://127.13.38.66:44413/transactions?tablet_id=a0e82ff0b52944bea2d643458a02405a
http://127.13.38.67:38021/scans
http://127.13.38.66:44413/scans
http://127.13.38.65:45799/threadz?group=all
http://127.13.38.126:37637/ipki-ca-cert-pem
http://127.13.38.67:38021/metrics
http://127.13.38.67:38021/stacks
http://127.13.38.66:44413/mem-trackers
http://127.13.38.67:38021/threadz?group=all
http://127.13.38.126:37637/ipki-ca-cert-der
http://127.13.38.65:45799/startup
http://127.13.38.67:38021/startup
http://127.13.38.66:44413/startup
http://127.13.38.126:37637/stacks
http://127.13.38.126:37637/mem-trackers
http://127.13.38.126:37637/startup
http://127.13.38.66:44413/tablets
http://127.13.38.126:37637/threadz?group=all
http://127.13.38.65:45799/scans
http://127.13.38.67:38021/maintenance-manager
http://127.13.38.65:45799/tablets
http://127.13.38.126:37637/tables
http://127.13.38.126:37637/masters
http://127.13.38.67:38021/transactions?tablet_id=a0e82ff0b52944bea2d643458a02405a
http://127.13.38.65:45799/stacks
http://127.13.38.66:44413/threadz
I20240426 10:00:34.704732 13465 linked_list-test-util.h:401] Running for: 30.000s
I20240426 10:00:34.704959 13465 linked_list-test-util.h:402] Sampling every 10000000 us
I20240426 10:00:34.705066 13465 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240426 10:00:34.759752 13780 raft_consensus.cc:1261] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Refusing update from remote peer 755ae40e4b834d9e8bdc3ae0cc726af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.759742 13649 raft_consensus.cc:1261] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Refusing update from remote peer 755ae40e4b834d9e8bdc3ae0cc726af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.760305 13910 raft_consensus.cc:1261] T a0e82ff0b52944bea2d643458a02405a P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Refusing update from remote peer 2491b640d5ae43ba955f47ab77ef7bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.760628 13979 consensus_queue.cc:1035] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.761199 13993 consensus_queue.cc:1035] T be24d5b36bce438abca45fe308c21b73 P 755ae40e4b834d9e8bdc3ae0cc726af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.761265 13983 consensus_queue.cc:1035] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.762703 13649 raft_consensus.cc:1261] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Refusing update from remote peer 66c3b02bff40431aaed66dba0405aed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.763584 13990 consensus_queue.cc:1035] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.765399 13779 raft_consensus.cc:1261] T a0e82ff0b52944bea2d643458a02405a P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Refusing update from remote peer 2491b640d5ae43ba955f47ab77ef7bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.766341 13910 raft_consensus.cc:1261] T 1c421916024e46a8bbe38590663e721e P 755ae40e4b834d9e8bdc3ae0cc726af2 [term 1 FOLLOWER]: Refusing update from remote peer 66c3b02bff40431aaed66dba0405aed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 10:00:34.766225 13997 consensus_queue.cc:1035] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.767369 13990 consensus_queue.cc:1035] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 10:00:34.797252 14008 mvcc.cc:205] Tried to move back new op lower bound from 7021058599964897280 to 7021058598820241408. Current Snapshot: MvccSnapshot[applied={T|T < 7021058599964897280}]
I20240426 10:00:34.798043 14007 mvcc.cc:205] Tried to move back new op lower bound from 7021058599968821248 to 7021058599328825344. Current Snapshot: MvccSnapshot[applied={T|T < 7021058599968821248}]
I20240426 10:00:34.815889 14009 mvcc.cc:205] Tried to move back new op lower bound from 7021058599964897280 to 7021058598820241408. Current Snapshot: MvccSnapshot[applied={T|T < 7021058599964897280}]
I20240426 10:00:36.027268 14010 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:36.029878 14001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:37.934712 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:40.105705 14010 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:40.111356 14001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:40.722689 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:40.735555 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:40.748271 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:43.078557 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 2.355s user 0.595s sys 0.114s Metrics: {"bytes_written":6653,"cfile_init":1,"compiler_manager_pool.queue_time_us":165785,"compiler_manager_pool.run_cpu_time_us":726398,"compiler_manager_pool.run_wall_time_us":1883650,"delete_count":0,"dirs.queue_time_us":2673,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":16702,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":56446,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7896,"thread_start_us":14933,"threads_started":2,"update_count":1009}
I20240426 10:00:43.080847 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 132810 bytes on disk
I20240426 10:00:43.091037 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240426 10:00:43.097783 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73): perf score=0.053541
I20240426 10:00:43.254056 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 2.505s user 0.615s sys 0.054s Metrics: {"bytes_written":6840,"cfile_init":1,"compiler_manager_pool.queue_time_us":21631,"compiler_manager_pool.run_cpu_time_us":702157,"compiler_manager_pool.run_wall_time_us":2148568,"delete_count":0,"dirs.queue_time_us":6402,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":11208,"drs_written":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2614,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7927,"thread_start_us":29507,"threads_started":2,"update_count":1040}
I20240426 10:00:43.264910 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 133134 bytes on disk
I20240426 10:00:43.267052 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240426 10:00:43.267963 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73): perf score=0.054780
I20240426 10:00:43.518234 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.420s user 0.060s sys 0.044s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7802,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":14918,"dirs.run_cpu_time_us":361,"dirs.run_wall_time_us":1904,"lbm_read_time_us":515,"lbm_reads_lt_1ms":20,"lbm_write_time_us":448,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":21424,"threads_started":2,"update_count":1009}
I20240426 10:00:43.584451 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 2.848s user 0.570s sys 0.147s Metrics: {"bytes_written":6840,"cfile_init":1,"compiler_manager_pool.queue_time_us":19290,"delete_count":0,"dirs.queue_time_us":3419,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":7696,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2461,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7927,"thread_start_us":23356,"threads_started":2,"update_count":1040}
I20240426 10:00:43.586575 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:43.587181 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73): perf score=0.054780
I20240426 10:00:43.644173 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.376s user 0.111s sys 0.012s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7992,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":32136,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":4113,"lbm_read_time_us":4099,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":19,"lbm_write_time_us":560,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":55168,"threads_started":1,"update_count":1040}
I20240426 10:00:43.806581 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: MajorDeltaCompactionOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.217s user 0.100s sys 0.025s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7992,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2106,"dirs.run_cpu_time_us":379,"dirs.run_wall_time_us":1675,"lbm_read_time_us":1513,"lbm_reads_lt_1ms":24,"lbm_write_time_us":434,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":37620,"threads_started":2,"update_count":1040}
I20240426 10:00:43.807523 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 133134 bytes on disk
I20240426 10:00:43.810191 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":525,"lbm_reads_lt_1ms":4}
I20240426 10:00:44.722823 13465 linked_list-test-util.h:417] Sample at HT timestamp: 7021058640709545984 Inserted count: 17500
I20240426 10:00:45.651180 14010 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:45.654307 14001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:47.439355 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:47.783471 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:47.819953 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:47.909052 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:49.018558 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.197s user 0.466s sys 0.109s Metrics: {"bytes_written":4804,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5795,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":6657,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2337,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7803,"thread_start_us":5967,"threads_started":1,"update_count":720}
I20240426 10:00:49.023841 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014688
I20240426 10:00:49.029186 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.245s user 0.538s sys 0.093s Metrics: {"bytes_written":4582,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":39648,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1882,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2497,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7800,"thread_start_us":39539,"threads_started":1,"update_count":686}
I20240426 10:00:49.030983 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 133038 bytes on disk
I20240426 10:00:49.032287 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240426 10:00:49.033231 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014689
I20240426 10:00:49.134663 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.225s user 0.516s sys 0.077s Metrics: {"bytes_written":5231,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":3927,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":4696,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2524,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7868,"thread_start_us":4674,"threads_started":1,"update_count":785}
I20240426 10:00:49.136512 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 133480 bytes on disk
I20240426 10:00:49.139029 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240426 10:00:49.140070 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014688
I20240426 10:00:50.063918 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.919s user 0.674s sys 0.066s Metrics: {"bytes_written":291184,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93933,"cfile_cache_miss":62,"cfile_cache_miss_bytes":962243,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":64754,"dirs.run_cpu_time_us":1050,"dirs.run_wall_time_us":9949,"drs_written":1,"lbm_read_time_us":3219,"lbm_reads_lt_1ms":98,"lbm_write_time_us":3710,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":518937,"rows_written":15795,"thread_start_us":70455,"threads_started":3}
I20240426 10:00:50.065472 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 289428 bytes on disk
I20240426 10:00:50.067070 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240426 10:00:50.238255 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.200s user 0.682s sys 0.055s Metrics: {"bytes_written":289620,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93453,"cfile_cache_miss":62,"cfile_cache_miss_bytes":956383,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":26971,"dirs.run_cpu_time_us":1180,"dirs.run_wall_time_us":6949,"drs_written":1,"lbm_read_time_us":3836,"lbm_reads_lt_1ms":98,"lbm_write_time_us":18935,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":49,"mutex_wait_us":18194,"num_input_rowsets":2,"peak_mem_usage":515629,"rows_written":15696,"thread_start_us":30800,"threads_started":3}
I20240426 10:00:50.239387 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 287776 bytes on disk
I20240426 10:00:50.240480 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240426 10:00:50.685222 14010 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:50.687572 14001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:50.786535 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.762s user 0.833s sys 0.002s Metrics: {"bytes_written":290156,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93933,"cfile_cache_miss":62,"cfile_cache_miss_bytes":958157,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":9080,"dirs.run_cpu_time_us":1083,"dirs.run_wall_time_us":13764,"drs_written":1,"lbm_read_time_us":5257,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":3921,"lbm_writes_lt_1ms":50,"mutex_wait_us":194,"num_input_rowsets":2,"peak_mem_usage":516762,"rows_written":15730,"thread_start_us":17889,"threads_started":3}
I20240426 10:00:50.787881 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 288335 bytes on disk
I20240426 10:00:50.788853 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240426 10:00:52.466674 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:54.576015 14010 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:54.582442 14001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 10:00:54.707830 13465 linked_list-test-util.h:417] Sample at HT timestamp: 7021058681640665088 Inserted count: 34600
I20240426 10:00:55.000929 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:55.057372 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:55.088483 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling FlushMRSOp(be24d5b36bce438abca45fe308c21b73): perf score=1.000000
I20240426 10:00:56.188189 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.187s user 0.471s sys 0.034s Metrics: {"bytes_written":4681,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":7192,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":5108,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3348,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7814,"thread_start_us":7364,"threads_started":1,"update_count":699}
I20240426 10:00:56.202428 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 133509 bytes on disk
I20240426 10:00:56.211710 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240426 10:00:56.220355 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014710
I20240426 10:00:56.281424 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.192s user 0.504s sys 0.022s Metrics: {"bytes_written":5115,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":8027,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2946,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2540,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7784,"thread_start_us":7903,"threads_started":1,"update_count":768}
I20240426 10:00:56.298316 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 132393 bytes on disk
I20240426 10:00:56.304755 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":179,"lbm_reads_lt_1ms":4}
I20240426 10:00:56.305969 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014709
I20240426 10:00:56.570713 13847 maintenance_manager.cc:643] P 755ae40e4b834d9e8bdc3ae0cc726af2: FlushMRSOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.513s user 0.481s sys 0.021s Metrics: {"bytes_written":4912,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":17599,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":2283,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2515,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7815,"thread_start_us":17756,"threads_started":1,"update_count":734}
I20240426 10:00:56.573768 13956 maintenance_manager.cc:419] P 755ae40e4b834d9e8bdc3ae0cc726af2: Scheduling CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73): perf score=1.014709
I20240426 10:00:56.795543 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240426 10:00:56.812038 14010 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fa51111ddbd google::LogMessage::Fail() at ??:0
@ 0x7fa511121b83 google::LogMessage::SendToLog() at ??:0
@ 0x7fa51111d7bc google::LogMessage::Flush() at ??:0
@ 0x7fa51111ef49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa513d45880 kudu::operator-() at ??:0
@ 0x7fa5209b739a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fa52098182e kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fa520981586 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fa52092e529 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fa5233f80a7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fa523406c5c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fa523406ac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa51ce476aa std::function<>::operator()() at ??:0
@ 0x7fa51ce3fe66 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fa51ce3ae71 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fa51ce38fa7 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fa51ce76709 handle_request at ??:0
@ 0x7fa51ce78deb worker_thread at ??:0
@ 0x7fa51718e6db start_thread at ??:0
@ 0x7fa50ffd071f clone at ??:0
W20240426 10:00:57.175320 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: recv got EOF from 127.13.38.67:37307 (error 108)
W20240426 10:00:57.183687 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.193147 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.228024 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.236554 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.266826 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.278458 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.309581 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.326267 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.361946 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.366727 13582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:00:57.375691 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 10:00:57.379419 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.390501 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:00:57.407272 13713 consensus_peers.cc:485] T 1c421916024e46a8bbe38590663e721e P 66c3b02bff40431aaed66dba0405aed7 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 10:00:57.422727 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.449440 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.501057 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.527272 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:00:57.568121 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.254s user 1.005s sys 0.044s Metrics: {"bytes_written":429000,"cfile_cache_hit":11,"cfile_cache_hit_bytes":802,"cfile_cache_miss":77,"cfile_cache_miss_bytes":1598458,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":7808,"dirs.run_cpu_time_us":825,"dirs.run_wall_time_us":13225,"drs_written":1,"lbm_read_time_us":9751,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":108,"lbm_write_time_us":5320,"lbm_writes_lt_1ms":62,"mutex_wait_us":34,"num_input_rowsets":2,"peak_mem_usage":779712,"rows_written":23579,"thread_start_us":8240,"threads_started":5}
I20240426 10:00:57.569191 13695 maintenance_manager.cc:419] P 2491b640d5ae43ba955f47ab77ef7bb2: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 438394 bytes on disk
I20240426 10:00:57.569738 13586 maintenance_manager.cc:643] P 2491b640d5ae43ba955f47ab77ef7bb2: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240426 10:00:57.580286 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.593475 13739 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.597738 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.608170 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.614661 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.621783 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.647383 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.658098 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.669911 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.684226 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.700850 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.702538 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.736546 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.755385 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:00:57.758312 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: CompactRowSetsOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 1.538s user 0.970s sys 0.023s Metrics: {"bytes_written":427660,"cfile_cache_hit":13,"cfile_cache_hit_bytes":181751,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1413525,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":9326,"dirs.run_cpu_time_us":1066,"dirs.run_wall_time_us":14794,"drs_written":1,"lbm_read_time_us":4119,"lbm_reads_lt_1ms":107,"lbm_write_time_us":5615,"lbm_writes_lt_1ms":62,"mutex_wait_us":93,"num_input_rowsets":2,"peak_mem_usage":777399,"rows_written":23510,"thread_start_us":10896,"threads_started":5}
I20240426 10:00:57.759550 13826 maintenance_manager.cc:419] P 66c3b02bff40431aaed66dba0405aed7: Scheduling UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73): 437123 bytes on disk
I20240426 10:00:57.760243 13717 maintenance_manager.cc:643] P 66c3b02bff40431aaed66dba0405aed7: UndoDeltaBlockGCOp(be24d5b36bce438abca45fe308c21b73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240426 10:00:57.769413 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.793391 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.802904 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.816390 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.862639 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.872465 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.887722 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.911185 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.938202 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.969537 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:57.990017 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.029671 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.030802 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.060549 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.117421 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.126201 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.158974 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.159816 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.197499 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:00:58.229807 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.250627 13740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33904: Illegal state: replica 66c3b02bff40431aaed66dba0405aed7 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.265281 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:00:58.290908 14238 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755ae40e4b834d9e8bdc3ae0cc726af2)
I20240426 10:00:58.291177 14238 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:58.292289 14238 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
I20240426 10:00:58.292949 13779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 771 } } ignore_live_leader: false dest_uuid: "66c3b02bff40431aaed66dba0405aed7" is_pre_election: true
W20240426 10:00:58.293998 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:00:58.294152 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:00:58.294485 14238 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:00:58.295390 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:00:58.295964 14214 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755ae40e4b834d9e8bdc3ae0cc726af2)
I20240426 10:00:58.296171 14214 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:58.297168 14214 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945)
I20240426 10:00:58.297751 13647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 2 candidate_status { last_received { term: 1 index: 771 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" is_pre_election: true
I20240426 10:00:58.298002 13647 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 1.
I20240426 10:00:58.298570 13713 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 66c3b02bff40431aaed66dba0405aed7; no voters:
I20240426 10:00:58.298877 14214 raft_consensus.cc:2781] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 10:00:58.298887 13713 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:00:58.298992 14214 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755ae40e4b834d9e8bdc3ae0cc726af2)
I20240426 10:00:58.299101 14214 raft_consensus.cc:3037] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 1 FOLLOWER]: Advancing to term 2
I20240426 10:00:58.301211 14214 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:58.301873 14214 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 election: Requested vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 2491b640d5ae43ba955f47ab77ef7bb2 (127.13.38.65:44945)
I20240426 10:00:58.302541 13647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be24d5b36bce438abca45fe308c21b73" candidate_uuid: "66c3b02bff40431aaed66dba0405aed7" candidate_term: 2 candidate_status { last_received { term: 1 index: 771 } } ignore_live_leader: false dest_uuid: "2491b640d5ae43ba955f47ab77ef7bb2"
I20240426 10:00:58.302739 13647 raft_consensus.cc:3037] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Advancing to term 2
W20240426 10:00:58.303368 13713 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:00:58.304966 13647 raft_consensus.cc:2454] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c3b02bff40431aaed66dba0405aed7 in term 2.
I20240426 10:00:58.305332 13713 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2, 66c3b02bff40431aaed66dba0405aed7; no voters: 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:00:58.305604 14214 raft_consensus.cc:2781] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 10:00:58.305758 14214 raft_consensus.cc:686] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [term 2 LEADER]: Becoming Leader. State: Replica: 66c3b02bff40431aaed66dba0405aed7, State: Running, Role: LEADER
I20240426 10:00:58.306068 14214 consensus_queue.cc:237] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 771, Committed index: 771, Last appended: 1.771, Last appended by leader: 771, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:00:58.308552 13506 catalog_manager.cc:5484] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 reported cstate change: term changed from 1 to 2, leader changed from 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67) to 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66). New cstate: current_term: 2 leader_uuid: "66c3b02bff40431aaed66dba0405aed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20240426 10:00:58.346112 13647 raft_consensus.cc:1261] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Refusing update from remote peer 66c3b02bff40431aaed66dba0405aed7: Log matching property violated. Preceding OpId in replica: term: 1 index: 771. Preceding OpId from leader: term: 2 index: 773. (index mismatch)
W20240426 10:00:58.346926 13713 consensus_peers.cc:485] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 10:00:58.346971 14237 consensus_queue.cc:1035] T be24d5b36bce438abca45fe308c21b73 P 66c3b02bff40431aaed66dba0405aed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 772, Last known committed idx: 771, Time since last communication: 0.000s
F20240426 10:00:58.594242 14001 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f7fcda44dbd google::LogMessage::Fail() at ??:0
@ 0x7f7fcda48b83 google::LogMessage::SendToLog() at ??:0
@ 0x7f7fcda447bc google::LogMessage::Flush() at ??:0
@ 0x7f7fcda45f49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7fd066c880 kudu::operator-() at ??:0
@ 0x7f7fdd2de39a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f7fdd2a882e kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f7fdd2a8586 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f7fdd255529 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f7fdfd1f0a7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f7fdfd2dc5c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f7fdfd2dac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7fd976e6aa std::function<>::operator()() at ??:0
@ 0x7f7fd9766e66 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f7fd9761e71 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f7fd975ffa7 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f7fd979d709 handle_request at ??:0
@ 0x7f7fd979fdeb worker_thread at ??:0
@ 0x7f7fd3ab56db start_thread at ??:0
@ 0x7f7fcc8f771f clone at ??:0
W20240426 10:00:58.881278 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.901880 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.936527 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:58.981968 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.031289 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.094484 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.166585 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.212172 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 10:00:59.218936 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:00:59.247115 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.339282 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.381536 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.384523 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:00:59.404438 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:00:59.404546 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.406695 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.411670 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.455274 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.484709 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.484763 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.516697 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.553577 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.601649 13608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.598990 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.648509 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.662811 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.748669 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.749804 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.812043 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.909440 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.925587 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 10:00:59.926856 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:00:59.978844 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.054170 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:00.087924 14172 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:00.088138 14172 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:00.088867 14172 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:00.090327 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:00.090773 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:00.090929 13584 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:00.091241 14172 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 10:01:00.100834 14172 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:00.101047 14172 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:00.101760 14172 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:00.103102 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:00.103355 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:00.103518 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:00.103825 14172 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:00.121376 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.228227 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.231405 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:00.251515 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.336449 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.447682 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.452055 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.475718 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:00.532366 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:00.562300 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.686765 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.718953 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.805361 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.812882 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:00.880292 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.004211 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.054067 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.136922 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.236625 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.276391 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:01.344461 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.422869 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.560144 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:01.657212 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:01.692595 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 10:01:01.693398 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:01.775326 14172 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:01.775552 14172 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:01.776276 14172 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:01.777984 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:01.778249 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:01.778403 13584 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:01.778702 14172 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:01.784648 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:01.903133 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.022050 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:02.060231 14172 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:02.060519 14172 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:02.061712 14172 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:02.063241 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:02.063537 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:02.063679 13584 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:02.063982 14172 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:02.124903 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.268419 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.308979 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:02.436141 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.494521 13582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240426 10:01:02.495596 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 10:01:02.502832 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.610783 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.743963 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:02.769240 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:02.876353 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:02.994379 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.147662 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.251788 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.337332 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:03.520459 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.531219 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.705153 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:03.824438 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:03.861266 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:03.990934 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:04.124840 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:04.206924 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 10:01:04.284041 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:04.341814 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:04.439064 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:04.591215 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:04.638206 14172 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:04.638480 14172 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:04.639499 14172 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:04.641184 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:04.641393 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:04.641521 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:04.641872 14172 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:04.760071 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:04.768780 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 10:01:04.902763 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:05.034193 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:05.088786 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:05.209740 14172 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:05.210000 14172 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:05.210739 14172 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:05.212319 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:05.212601 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:05.212718 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:05.213013 14172 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:05.221683 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:05.388900 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:05.426090 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:05.551733 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:05.776228 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.004957 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.135177 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:06.136488 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:06.360424 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.416654 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:06.502799 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.722249 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.877075 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:06.980465 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 10:01:07.023478 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 10:01:07.087069 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:07.220482 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:07.259765 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:07.468390 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:07.503674 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240426 10:01:07.519119 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:07.649199 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:07.856894 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:08.051867 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:08.169978 14172 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:08.170217 14172 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:08.171924 14172 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:08.173102 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:08.173311 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:08.173435 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:08.173709 14172 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:08.254185 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:08.311342 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:08.467463 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:08.527030 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:08.660038 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:08.866365 14172 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:08.866564 14172 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:08.867292 14172 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:08.869094 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:08.869320 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:08.869460 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:08.869722 14172 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:08.888729 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:09.079180 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:09.314378 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:09.386493 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 10:01:09.399996 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 10:01:09.412876 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:09.502501 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:09.543429 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:09.751833 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:09.932134 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:10.203243 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:10.378975 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:10.485708 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:10.561486 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:10.654929 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:10.826218 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:11.120071 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:11.282631 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:11.543545 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:11.595170 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:11.596745 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:11.611435 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240426 10:01:11.751299 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:11.916185 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:12.019848 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 10:01:12.227275 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:12.402586 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:12.504150 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 24 similar messages]
I20240426 10:01:12.584344 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:12.598294 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:12.713698 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:12.901476 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:13.210476 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:13.405699 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:13.659265 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:13.715121 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:13.746196 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:13.768173 14172 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:13.768440 14172 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:13.769225 14172 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:13.771817 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:13.772138 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:13.772258 13584 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:13.772593 14172 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:13.920224 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:14.228672 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:14.229737 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240426 10:01:14.441785 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:14.491490 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240426 10:01:14.723863 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:14.750547 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:14.793884 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:14.976326 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:15.279457 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:15.517962 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:15.734427 14172 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:15.734647 14172 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:15.735339 14172 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:15.736949 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:15.737285 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:15.737419 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:15.737684 14172 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 10:01:15.801225 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:15.815158 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:15.869772 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:16.070241 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:16.362507 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:16.632730 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:16.700294 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 10:01:16.716276 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240426 10:01:16.866204 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:16.918797 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:16.976275 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:17.202490 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:17.480934 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:17.779902 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:17.800100 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 24 similar messages]
I20240426 10:01:17.947929 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:18.056854 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:18.117547 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:18.362320 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:18.643249 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:18.959806 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:18.995695 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:19.158867 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:19.200523 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240426 10:01:19.238306 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:19.379259 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 10:01:19.562638 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:19.842535 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:20.078622 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:20.177287 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:20.247225 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:20.452061 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:20.796604 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:21.073521 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:21.128870 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:21.281281 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:21.426352 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:21.468245 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240426 10:01:21.621937 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240426 10:01:21.701107 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:22.066937 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:22.164561 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:22.282738 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:22.340824 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:22.423096 14292 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:22.423318 14292 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:22.424073 14292 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:22.425753 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:22.426043 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:22.426213 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:22.426540 14292 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:22.718619 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:22.830420 13582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240426 10:01:22.993021 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:23.193097 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:23.376749 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:23.428015 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:23.645457 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:23.844206 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240426 10:01:23.976832 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240426 10:01:24.046514 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:24.209319 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:24.307021 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:24.498749 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:24.722581 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:24.978664 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:25.212713 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:25.412218 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:25.420002 14292 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:25.420209 14292 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:25.420888 14292 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:25.422453 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:25.422664 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:25.422833 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:25.423175 14292 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:25.640933 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:25.658982 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:26.106284 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:26.203281 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240426 10:01:26.279335 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:26.344504 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240426 10:01:26.352670 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:26.644235 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:26.812634 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:27.286592 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:27.315305 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:27.531438 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:27.764864 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:27.913388 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 10:01:28.002295 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:28.246464 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:28.396358 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:28.723799 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:28.779361 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:28.795205 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240426 10:01:28.825287 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 10:01:28.977686 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:29.447810 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:29.460410 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:29.721289 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:29.955144 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:30.201933 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:30.468807 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:30.495692 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:30.949954 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:30.976117 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:31.229772 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:31.264415 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240426 10:01:31.338956 14292 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:31.339226 14292 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:31.340262 14292 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:31.341912 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:31.342283 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:31.342389 13584 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:31.342646 14292 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:31.434170 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240426 10:01:31.543292 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:31.702265 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:31.995940 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:32.014171 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:32.466986 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:32.584362 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:32.769898 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:33.029151 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:33.241861 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 10:01:33.247658 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:33.553512 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:33.602042 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:33.678074 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240426 10:01:33.849413 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240426 10:01:34.026224 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:34.080756 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:34.347622 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:34.620448 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:34.819244 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:35.084060 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:35.146883 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:35.621064 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:35.641705 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:35.913411 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240426 10:01:35.956928 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:36.159718 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:36.341041 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240426 10:01:36.430218 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:36.633107 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:36.780824 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:37.252290 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:37.307590 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:37.610054 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:37.645275 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:38.084895 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:38.362748 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:38.449136 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:38.449913 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240426 10:01:38.451032 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240426 10:01:38.679816 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:38.924175 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:38.991549 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240426 10:01:39.088398 14298 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:39.088589 14298 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:39.089212 14298 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:39.090852 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:39.090986 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:39.091090 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:39.091332 14298 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:39.300559 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:39.485760 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:39.709604 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:39.770193 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:40.156821 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:40.627621 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:40.680181 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:40.756951 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:41.027428 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:41.150238 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240426 10:01:41.472468 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240426 10:01:41.495836 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:41.784560 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:41.785486 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:41.908993 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:42.366655 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:42.795195 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:42.813289 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:42.955713 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:43.248733 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:43.691071 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:43.819327 13582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240426 10:01:43.820199 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240426 10:01:43.828676 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:43.992516 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:44.023903 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240426 10:01:44.143720 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:44.594002 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:44.834102 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:45.045317 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:45.098992 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:45.507478 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:45.841006 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:45.955699 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:46.117895 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:46.133822 14302 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:46.134090 14302 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:46.135004 14302 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:46.136235 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:46.136559 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:46.136659 13582 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:46.136901 14302 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 10:01:46.372697 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240426 10:01:46.424268 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:46.622493 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240426 10:01:46.840688 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:46.876921 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:47.183773 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:47.354377 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:47.794436 14302 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:47.794684 14302 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:47.795632 14302 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:47.797194 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:47.797435 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:47.797590 13582 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:47.797922 14302 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:47.802484 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:47.873714 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:48.294564 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:48.423321 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:48.729032 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240426 10:01:48.734813 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:48.824183 13584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240426 10:01:48.825141 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240426 10:01:48.865327 14302 raft_consensus.cc:482] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:48.865500 14302 raft_consensus.cc:504] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } }
I20240426 10:01:48.866153 14302 leader_election.cc:288] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215), 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307)
W20240426 10:01:48.867619 13582 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:48.867825 13584 leader_election.cc:334] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:48.867915 13584 leader_election.cc:302] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:48.868155 14302 raft_consensus.cc:2726] T 1c421916024e46a8bbe38590663e721e P 2491b640d5ae43ba955f47ab77ef7bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 10:01:48.903205 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:49.239310 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:49.435246 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:49.681970 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:49.913017 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:50.197153 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:50.517976 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:50.640322 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:50.915167 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:51.161938 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:51.162755 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240426 10:01:51.457886 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240426 10:01:51.608222 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:51.678529 13958 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:51.927832 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:52.135207 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:52.585722 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:52.791476 13960 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:52.931006 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:53.121173 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:53.571055 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:53.790941 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240426 10:01:53.901382 13958 meta_cache.cc:302] tablet be24d5b36bce438abca45fe308c21b73: replica 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
I20240426 10:01:53.938410 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:53.998801 13582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240426 10:01:53.999778 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240426 10:01:54.110409 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:54.564764 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:54.943259 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:54.959000 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:55.110337 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:55.565248 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:55.929695 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:55.964877 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:56.119136 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:56.262360 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240426 10:01:56.544968 14302 raft_consensus.cc:482] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 66c3b02bff40431aaed66dba0405aed7)
I20240426 10:01:56.545218 14302 raft_consensus.cc:504] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "755ae40e4b834d9e8bdc3ae0cc726af2" member_type: VOTER last_known_addr { host: "127.13.38.67" port: 37307 } } peers { permanent_uuid: "66c3b02bff40431aaed66dba0405aed7" member_type: VOTER last_known_addr { host: "127.13.38.66" port: 46215 } } peers { permanent_uuid: "2491b640d5ae43ba955f47ab77ef7bb2" member_type: VOTER last_known_addr { host: "127.13.38.65" port: 44945 } }
I20240426 10:01:56.546191 14302 leader_election.cc:288] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307), 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215)
W20240426 10:01:56.547410 13582 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111)
W20240426 10:01:56.547703 13584 leader_election.cc:334] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
I20240426 10:01:56.547806 13584 leader_election.cc:302] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2491b640d5ae43ba955f47ab77ef7bb2; no voters: 66c3b02bff40431aaed66dba0405aed7, 755ae40e4b834d9e8bdc3ae0cc726af2
I20240426 10:01:56.548029 14302 raft_consensus.cc:2726] T be24d5b36bce438abca45fe308c21b73 P 2491b640d5ae43ba955f47ab77ef7bb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 10:01:56.571702 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240426 10:01:56.575042 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:56.946429 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:56.983512 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:57.136736 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:57.595103 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
I20240426 10:01:57.975112 14018 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 10:01:58.008385 13960 meta_cache.cc:302] tablet 1c421916024e46a8bbe38590663e721e: replica 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215) has failed: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111)
W20240426 10:01:58.165766 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:58.406450 13584 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 66c3b02bff40431aaed66dba0405aed7 (127.13.38.66:46215): Couldn't send request to peer 66c3b02bff40431aaed66dba0405aed7. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.66:46215: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240426 10:01:58.624521 13609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56610: Illegal state: replica 2491b640d5ae43ba955f47ab77ef7bb2 is not leader of this config: current role FOLLOWER
W20240426 10:01:58.799413 13582 consensus_peers.cc:485] T a0e82ff0b52944bea2d643458a02405a P 2491b640d5ae43ba955f47ab77ef7bb2 -> Peer 755ae40e4b834d9e8bdc3ae0cc726af2 (127.13.38.67:37307): Couldn't send request to peer 755ae40e4b834d9e8bdc3ae0cc726af2. Status: Network error: Client connection negotiation failed: client connection to 127.13.38.67:37307: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240426 10:01:58.855602 13958 batcher.cc:441] Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.856555 13465 client-test-util.cc:49] 35 failed ops. First 10 errors follow
I20240426 10:01:58.856726 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-9056722462877483008, int64 link_to=-2782356865902510080, int64 insert_ts=1714125658589220 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.856952 13465 client-test-util.cc:55] Op INSERT int64 rand_key=1289561817080856577, int64 link_to=-1427911165804871679, int64 insert_ts=1714125658589441 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857069 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-6811320189135355902, int64 link_to=-73043253242232830, int64 insert_ts=1714125658589505 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857146 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-3276637573288951803, int64 link_to=1208218456124882949, int64 insert_ts=1714125658589656 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857239 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-1031235299546824697, int64 link_to=3917391331296870407, int64 insert_ts=1714125658589744 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857311 13465 client-test-util.cc:55] Op INSERT int64 rand_key=258185780045742088, int64 link_to=2489480165491998728, int64 insert_ts=1714125658589789 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857400 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-7842414751193759735, int64 link_to=3844207340566347785, int64 insert_ts=1714125658589869 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857470 13465 client-test-util.cc:55] Op INSERT int64 rand_key=2503447316299579402, int64 link_to=5198653040663986186, int64 insert_ts=1714125658589937 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857556 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-5597293952428343285, int64 link_to=6553098740761624587, int64 insert_ts=1714125658589984 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
I20240426 10:01:58.857620 13465 client-test-util.cc:55] Op INSERT int64 rand_key=-4307591397859065844, int64 link_to=5125469049933463564, int64 insert_ts=1714125658590027 had status Timed out: Failed to write batch of 35 ops to tablet be24d5b36bce438abca45fe308c21b73 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: be24d5b36bce438abca45fe308c21b73, num_ops: 35, num_attempts: 342) passed its deadline: Network error: recv got EOF from 127.13.38.66:46215 (error 108)
F20240426 10:01:58.857707 13465 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714125718 (unix time) try "date -d @1714125718" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003499) received by PID 13465 (TID 0x7fef17426d80) from PID 13465; stack trace: ***
@ 0x7fef1ee2e980 (unknown) at ??:0
@ 0x7fef17edefb7 gsignal at ??:0
@ 0x7fef17ee0921 abort at ??:0
@ 0x7fef1ac06dbd google::LogMessage::Fail() at ??:0
@ 0x7fef1ac0ab83 google::LogMessage::SendToLog() at ??:0
@ 0x7fef1ac067bc google::LogMessage::Flush() at ??:0
@ 0x7fef1ac07f49 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fef2a689ebd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed282 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/client/client-test-util.h:?
@ 0x3d5f48 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8249 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7fef1caa00ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fef1ca94bdb testing::Test::Run() at ??:0
@ 0x7fef1ca94d9d testing::TestInfo::Run() at ??:0
@ 0x7fef1ca95377 testing::TestSuite::Run() at ??:0
@ 0x7fef1ca9577c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fef1caa060d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fef1ca94e63 testing::UnitTest::Run() at ??:0
@ 0x7fef31768272 RUN_ALL_TESTS() at ??:0
@ 0x7fef31766789 main at ??:0
@ 0x7fef17ec1bf7 __libc_start_main at ??:0
@ 0x2eb91a _start at ??:?
E20240426 10:01:59.110921 14006 webserver.cc:533] Webserver: error reading: Connection reset by peer