[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 10:27:38.291764 1276 test_util.cc:268] Using random seed: -776860153
I20240430 10:27:38.303762 1276 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 10:27:38.303848 1276 linked_list-test.cc:101] --------------
I20240430 10:27:38.303887 1276 linked_list-test.cc:102] 50 chains
I20240430 10:27:38.303951 1276 linked_list-test.cc:103] 3 tablets
I20240430 10:27:38.304020 1276 linked_list-test.cc:104] Mutations on
I20240430 10:27:38.304090 1276 linked_list-test.cc:105] --------------
I20240430 10:27:38.304320 1276 ts_itest-base.cc:112] Starting cluster with:
I20240430 10:27:38.304378 1276 ts_itest-base.cc:113] --------------
I20240430 10:27:38.304438 1276 ts_itest-base.cc:114] 3 tablet servers
I20240430 10:27:38.304492 1276 ts_itest-base.cc:115] 3 replicas per TS
I20240430 10:27:38.304553 1276 ts_itest-base.cc:116] --------------
2024-04-30T10:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T10:27:38Z Disabled control of system clock
I20240430 10:27:38.328646 1276 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5w89pn/build/asan/bin/kudu
/tmp/dist-test-task5w89pn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.63.62:41335
--webserver_interface=127.1.63.62
--webserver_port=0
--builtin_ntp_servers=127.1.63.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.63.62:41335
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 10:27:38.673091 1288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:27:38.673436 1288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:27:38.673592 1288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:27:38.681622 1288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 10:27:38.681767 1288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:27:38.681862 1288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 10:27:38.681954 1288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 10:27:38.690963 1288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.63.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.63.62:41335
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.63.62:41335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.1.63.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 10:14:04 UTC on efd1337a2aa3
build id 55
ASAN enabled
I20240430 10:27:38.692391 1288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:27:38.694263 1288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:27:38.706328 1293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:38.707274 1294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:38.708089 1296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:27:38.708940 1288 server_base.cc:1034] running on GCE node
I20240430 10:27:38.709748 1288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:27:38.711161 1288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:27:38.712375 1288 hybrid_clock.cc:648] HybridClock initialized: now 1714472858712318 us; error 34 us; skew 500 ppm
I20240430 10:27:38.712713 1288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 10:27:38.721258 1288 webserver.cc:460] Webserver started at http://127.1.63.62:37905/ using document root <none> and password file <none>
I20240430 10:27:38.722311 1288 fs_manager.cc:359] Metadata directory not provided
I20240430 10:27:38.722489 1288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:27:38.722945 1288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:27:38.726976 1288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data/instance:
uuid: "06c82d9b20e447468b2cdf4ae71084dd"
format_stamp: "Formatted at 2024-04-30 10:27:38 on dist-test-slave-cw3g"
I20240430 10:27:38.728053 1288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal/instance:
uuid: "06c82d9b20e447468b2cdf4ae71084dd"
format_stamp: "Formatted at 2024-04-30 10:27:38 on dist-test-slave-cw3g"
I20240430 10:27:38.735406 1288 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20240430 10:27:38.739943 1302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:38.741771 1288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 10:27:38.741998 1288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data,/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal
uuid: "06c82d9b20e447468b2cdf4ae71084dd"
format_stamp: "Formatted at 2024-04-30 10:27:38 on dist-test-slave-cw3g"
I20240430 10:27:38.742250 1288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:27:38.757925 1288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 10:27:38.758790 1288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:27:38.759076 1288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:27:38.778313 1288 rpc_server.cc:307] RPC server started. Bound to: 127.1.63.62:41335
I20240430 10:27:38.778332 1353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.63.62:41335 every 8 connection(s)
I20240430 10:27:38.779673 1288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/data/info.pb
I20240430 10:27:38.783267 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:38.786724 1276 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5w89pn/build/asan/bin/kudu as pid 1288
I20240430 10:27:38.787012 1276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/master-0/wal/instance
I20240430 10:27:38.795909 1354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd: Bootstrap starting.
I20240430 10:27:38.798884 1354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:38.800561 1354 log.cc:824] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd: Log is configured to *not* fsync() on all Append() calls
I20240430 10:27:38.803191 1354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd: No bootstrap required, opened a new log
I20240430 10:27:38.810356 1354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } }
I20240430 10:27:38.810676 1354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:38.810772 1354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c82d9b20e447468b2cdf4ae71084dd, State: Initialized, Role: FOLLOWER
I20240430 10:27:38.811386 1354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } }
I20240430 10:27:38.811592 1354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 10:27:38.811710 1354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 10:27:38.811863 1354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:38.814220 1354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } }
I20240430 10:27:38.814682 1354 leader_election.cc:302] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c82d9b20e447468b2cdf4ae71084dd; no voters:
I20240430 10:27:38.815330 1354 leader_election.cc:288] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 10:27:38.815474 1359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:27:38.817447 1354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [sys.catalog]: configured and running, proceeding with master startup.
I20240430 10:27:38.817576 1359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [term 1 LEADER]: Becoming Leader. State: Replica: 06c82d9b20e447468b2cdf4ae71084dd, State: Running, Role: LEADER
I20240430 10:27:38.818094 1359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } }
I20240430 10:27:38.821411 1361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c82d9b20e447468b2cdf4ae71084dd. Latest consensus state: current_term: 1 leader_uuid: "06c82d9b20e447468b2cdf4ae71084dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } } }
I20240430 10:27:38.821709 1361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [sys.catalog]: This master's current role is: LEADER
I20240430 10:27:38.822247 1360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c82d9b20e447468b2cdf4ae71084dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c82d9b20e447468b2cdf4ae71084dd" member_type: VOTER last_known_addr { host: "127.1.63.62" port: 41335 } } }
I20240430 10:27:38.822590 1360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd [sys.catalog]: This master's current role is: LEADER
I20240430 10:27:38.824800 1367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 10:27:38.832144 1367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 10:27:38.841645 1367 catalog_manager.cc:1347] Generated new cluster ID: 68370b02d78949919a8f90ad92ad5996
I20240430 10:27:38.841763 1367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 10:27:38.860944 1367 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 10:27:38.862674 1367 catalog_manager.cc:1504] Loading token signing keys...
I20240430 10:27:38.873289 1367 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06c82d9b20e447468b2cdf4ae71084dd: Generated new TSK 0
I20240430 10:27:38.874163 1367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 10:27:38.889318 1276 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5w89pn/build/asan/bin/kudu
/tmp/dist-test-task5w89pn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.63.1:0
--local_ip_for_outbound_sockets=127.1.63.1
--webserver_interface=127.1.63.1
--webserver_port=0
--tserver_master_addrs=127.1.63.62:41335
--builtin_ntp_servers=127.1.63.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 10:27:39.248822 1378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:27:39.249122 1378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:27:39.249246 1378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:27:39.257081 1378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:27:39.257221 1378 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:27:39.257264 1378 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:27:39.257323 1378 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:27:39.257401 1378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.63.1
I20240430 10:27:39.266417 1378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.63.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.63.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.1.63.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.63.62:41335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.63.1
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 10:14:04 UTC on efd1337a2aa3
build id 55
ASAN enabled
I20240430 10:27:39.267724 1378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:27:39.269412 1378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:27:39.281955 1383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:39.282752 1386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:39.283452 1384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:27:39.284376 1378 server_base.cc:1034] running on GCE node
I20240430 10:27:39.284953 1378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:27:39.285892 1378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:27:39.287132 1378 hybrid_clock.cc:648] HybridClock initialized: now 1714472859287080 us; error 40 us; skew 500 ppm
I20240430 10:27:39.287442 1378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 10:27:39.289726 1378 webserver.cc:460] Webserver started at http://127.1.63.1:34041/ using document root <none> and password file <none>
I20240430 10:27:39.290506 1378 fs_manager.cc:359] Metadata directory not provided
I20240430 10:27:39.290621 1378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:27:39.290920 1378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:27:39.294090 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data/instance:
uuid: "0e59bbbfd1aa4f4898d18898f216ce58"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.294857 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal/instance:
uuid: "0e59bbbfd1aa4f4898d18898f216ce58"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.300143 1378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240430 10:27:39.303545 1392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.304857 1378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 10:27:39.305016 1378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal
uuid: "0e59bbbfd1aa4f4898d18898f216ce58"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.305210 1378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:27:39.329502 1378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 10:27:39.330353 1378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:27:39.330631 1378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:27:39.331578 1378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:27:39.333089 1378 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:27:39.333184 1378 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.333303 1378 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:27:39.333374 1378 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.368364 1378 rpc_server.cc:307] RPC server started. Bound to: 127.1.63.1:39971
I20240430 10:27:39.368446 1504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.63.1:39971 every 8 connection(s)
I20240430 10:27:39.369655 1378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/data/info.pb
I20240430 10:27:39.378988 1276 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5w89pn/build/asan/bin/kudu as pid 1378
I20240430 10:27:39.379202 1276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-0/wal/instance
I20240430 10:27:39.382539 1276 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5w89pn/build/asan/bin/kudu
/tmp/dist-test-task5w89pn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.63.2:0
--local_ip_for_outbound_sockets=127.1.63.2
--webserver_interface=127.1.63.2
--webserver_port=0
--tserver_master_addrs=127.1.63.62:41335
--builtin_ntp_servers=127.1.63.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 10:27:39.383277 1505 heartbeater.cc:347] Connected to a master server at 127.1.63.62:41335
I20240430 10:27:39.383565 1505 heartbeater.cc:464] Registering TS with master...
I20240430 10:27:39.384388 1505 heartbeater.cc:511] Master 127.1.63.62:41335 requested a full tablet report, sending...
I20240430 10:27:39.386401 1319 ts_manager.cc:194] Registered new tserver with Master: 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971)
I20240430 10:27:39.387868 1319 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.63.1:46169
W20240430 10:27:39.726644 1509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:27:39.726951 1509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:27:39.727113 1509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:27:39.734714 1509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:27:39.734889 1509 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:27:39.734958 1509 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:27:39.735029 1509 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:27:39.735136 1509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.63.2
I20240430 10:27:39.744652 1509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.63.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.63.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.1.63.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.63.62:41335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.63.2
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 10:14:04 UTC on efd1337a2aa3
build id 55
ASAN enabled
I20240430 10:27:39.746004 1509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:27:39.747853 1509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:27:39.760864 1514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:39.761598 1515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:39.762329 1517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:27:39.763212 1509 server_base.cc:1034] running on GCE node
I20240430 10:27:39.763909 1509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:27:39.764955 1509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:27:39.766180 1509 hybrid_clock.cc:648] HybridClock initialized: now 1714472859766137 us; error 41 us; skew 500 ppm
I20240430 10:27:39.766489 1509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 10:27:39.768811 1509 webserver.cc:460] Webserver started at http://127.1.63.2:42587/ using document root <none> and password file <none>
I20240430 10:27:39.769565 1509 fs_manager.cc:359] Metadata directory not provided
I20240430 10:27:39.769681 1509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:27:39.769990 1509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:27:39.773128 1509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data/instance:
uuid: "45b95ae9cf4a42c08032faef05080145"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.773869 1509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal/instance:
uuid: "45b95ae9cf4a42c08032faef05080145"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.779073 1509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240430 10:27:39.783478 1523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.784963 1509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 10:27:39.785127 1509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal
uuid: "45b95ae9cf4a42c08032faef05080145"
format_stamp: "Formatted at 2024-04-30 10:27:39 on dist-test-slave-cw3g"
I20240430 10:27:39.785334 1509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:27:39.804108 1509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 10:27:39.805070 1509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:27:39.805373 1509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:27:39.806596 1509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:27:39.808476 1509 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:27:39.808580 1509 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.808683 1509 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:27:39.808779 1509 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:39.840641 1509 rpc_server.cc:307] RPC server started. Bound to: 127.1.63.2:34979
I20240430 10:27:39.840683 1635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.63.2:34979 every 8 connection(s)
I20240430 10:27:39.841846 1509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/data/info.pb
I20240430 10:27:39.850426 1276 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5w89pn/build/asan/bin/kudu as pid 1509
I20240430 10:27:39.850589 1276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-1/wal/instance
I20240430 10:27:39.853811 1276 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5w89pn/build/asan/bin/kudu
/tmp/dist-test-task5w89pn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.63.3:0
--local_ip_for_outbound_sockets=127.1.63.3
--webserver_interface=127.1.63.3
--webserver_port=0
--tserver_master_addrs=127.1.63.62:41335
--builtin_ntp_servers=127.1.63.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 10:27:39.854427 1636 heartbeater.cc:347] Connected to a master server at 127.1.63.62:41335
I20240430 10:27:39.854707 1636 heartbeater.cc:464] Registering TS with master...
I20240430 10:27:39.856143 1636 heartbeater.cc:511] Master 127.1.63.62:41335 requested a full tablet report, sending...
I20240430 10:27:39.857488 1319 ts_manager.cc:194] Registered new tserver with Master: 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
I20240430 10:27:39.858155 1319 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.63.2:56353
W20240430 10:27:40.181811 1639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:27:40.182106 1639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:27:40.182243 1639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:27:40.189939 1639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:27:40.190073 1639 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:27:40.190111 1639 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:27:40.190191 1639 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:27:40.190280 1639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.63.3
I20240430 10:27:40.201053 1639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.63.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.63.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.1.63.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.63.62:41335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.63.3
--log_dir=/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 10:14:04 UTC on efd1337a2aa3
build id 55
ASAN enabled
I20240430 10:27:40.202316 1639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:27:40.203999 1639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:27:40.216444 1644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:40.216864 1647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:27:40.217681 1645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:27:40.222095 1639 server_base.cc:1034] running on GCE node
I20240430 10:27:40.222661 1639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:27:40.223596 1639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:27:40.224824 1639 hybrid_clock.cc:648] HybridClock initialized: now 1714472860224774 us; error 45 us; skew 500 ppm
I20240430 10:27:40.225134 1639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 10:27:40.227380 1639 webserver.cc:460] Webserver started at http://127.1.63.3:36649/ using document root <none> and password file <none>
I20240430 10:27:40.228168 1639 fs_manager.cc:359] Metadata directory not provided
I20240430 10:27:40.228289 1639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:27:40.228578 1639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:27:40.231122 1639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data/instance:
uuid: "3e663590b34d4d23889e203c7f2b7d41"
format_stamp: "Formatted at 2024-04-30 10:27:40 on dist-test-slave-cw3g"
I20240430 10:27:40.231815 1639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal/instance:
uuid: "3e663590b34d4d23889e203c7f2b7d41"
format_stamp: "Formatted at 2024-04-30 10:27:40 on dist-test-slave-cw3g"
I20240430 10:27:40.237121 1639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240430 10:27:40.240404 1653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:40.241750 1639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240430 10:27:40.241945 1639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal
uuid: "3e663590b34d4d23889e203c7f2b7d41"
format_stamp: "Formatted at 2024-04-30 10:27:40 on dist-test-slave-cw3g"
I20240430 10:27:40.242136 1639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:27:40.274581 1639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 10:27:40.275490 1639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:27:40.275776 1639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:27:40.277352 1639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:27:40.279456 1639 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:27:40.279551 1639 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:40.279667 1639 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:27:40.279735 1639 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 10:27:40.314370 1639 rpc_server.cc:307] RPC server started. Bound to: 127.1.63.3:41411
I20240430 10:27:40.314419 1765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.63.3:41411 every 8 connection(s)
I20240430 10:27:40.315656 1639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/data/info.pb
I20240430 10:27:40.320652 1276 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5w89pn/build/asan/bin/kudu as pid 1639
I20240430 10:27:40.320868 1276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5w89pn/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714472858210654-1276-0/linked-list-cluster/ts-2/wal/instance
I20240430 10:27:40.329097 1766 heartbeater.cc:347] Connected to a master server at 127.1.63.62:41335
I20240430 10:27:40.329362 1766 heartbeater.cc:464] Registering TS with master...
I20240430 10:27:40.330094 1766 heartbeater.cc:511] Master 127.1.63.62:41335 requested a full tablet report, sending...
I20240430 10:27:40.331385 1319 ts_manager.cc:194] Registered new tserver with Master: 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:27:40.332185 1319 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.63.3:34395
I20240430 10:27:40.337364 1276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:27:40.358489 1319 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41574:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240430 10:27:40.364583 1319 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 10:27:40.390533 1505 heartbeater.cc:503] Master 127.1.63.62:41335 was elected leader, sending a full tablet report...
I20240430 10:27:40.409828 1438 tablet_service.cc:1467] Processing CreateTablet for tablet a844d0da5c5143bbab129e5a8acee0ec (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:27:40.412335 1439 tablet_service.cc:1467] Processing CreateTablet for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:27:40.414247 1438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a844d0da5c5143bbab129e5a8acee0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.414279 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 612ef01002fd49128ee44f71a75c4690 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:27:40.415469 1439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.414283 1700 tablet_service.cc:1467] Processing CreateTablet for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:27:40.409794 1440 tablet_service.cc:1467] Processing CreateTablet for tablet 612ef01002fd49128ee44f71a75c4690 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:27:40.416007 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612ef01002fd49128ee44f71a75c4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.416548 1440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612ef01002fd49128ee44f71a75c4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.416545 1569 tablet_service.cc:1467] Processing CreateTablet for tablet a844d0da5c5143bbab129e5a8acee0ec (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:27:40.416893 1701 tablet_service.cc:1467] Processing CreateTablet for tablet 612ef01002fd49128ee44f71a75c4690 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:27:40.417071 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a844d0da5c5143bbab129e5a8acee0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.417665 1701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612ef01002fd49128ee44f71a75c4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.417819 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4 (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:27:40.418375 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.420234 1699 tablet_service.cc:1467] Processing CreateTablet for tablet a844d0da5c5143bbab129e5a8acee0ec (DEFAULT_TABLE table=TestTable [id=d66f62a37f794b18b604b90567ff7ef0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:27:40.420672 1699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a844d0da5c5143bbab129e5a8acee0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.434232 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f07b0d3c8b41bdbd4bef15ff4842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:27:40.449236 1784 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: Bootstrap starting.
I20240430 10:27:40.451617 1785 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Bootstrap starting.
I20240430 10:27:40.453274 1784 tablet_bootstrap.cc:654] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.454596 1784 log.cc:824] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: Log is configured to *not* fsync() on all Append() calls
I20240430 10:27:40.455503 1785 tablet_bootstrap.cc:654] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.456866 1785 log.cc:824] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Log is configured to *not* fsync() on all Append() calls
I20240430 10:27:40.457304 1786 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: Bootstrap starting.
I20240430 10:27:40.458307 1784 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: No bootstrap required, opened a new log
I20240430 10:27:40.458598 1784 ts_tablet_manager.cc:1367] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240430 10:27:40.459952 1785 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: No bootstrap required, opened a new log
I20240430 10:27:40.460254 1785 ts_tablet_manager.cc:1367] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240430 10:27:40.460805 1786 tablet_bootstrap.cc:654] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.461990 1786 log.cc:824] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: Log is configured to *not* fsync() on all Append() calls
I20240430 10:27:40.468528 1784 raft_consensus.cc:348] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.469079 1784 raft_consensus.cc:374] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.469197 1784 raft_consensus.cc:729] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b95ae9cf4a42c08032faef05080145, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.470119 1784 consensus_queue.cc:260] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.471375 1636 heartbeater.cc:503] Master 127.1.63.62:41335 was elected leader, sending a full tablet report...
I20240430 10:27:40.472175 1784 ts_tablet_manager.cc:1398] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145: Time spent starting tablet: real 0.013s user 0.016s sys 0.000s
I20240430 10:27:40.472636 1784 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145: Bootstrap starting.
I20240430 10:27:40.473367 1785 raft_consensus.cc:348] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.473845 1785 raft_consensus.cc:374] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.473982 1785 raft_consensus.cc:729] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e59bbbfd1aa4f4898d18898f216ce58, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.474735 1785 consensus_queue.cc:260] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.475894 1786 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: No bootstrap required, opened a new log
I20240430 10:27:40.476279 1786 ts_tablet_manager.cc:1367] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240430 10:27:40.476471 1785 ts_tablet_manager.cc:1398] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20240430 10:27:40.476866 1785 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58: Bootstrap starting.
I20240430 10:27:40.479159 1785 tablet_bootstrap.cc:654] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.481491 1785 tablet_bootstrap.cc:492] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58: No bootstrap required, opened a new log
I20240430 10:27:40.481631 1785 ts_tablet_manager.cc:1367] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240430 10:27:40.482038 1784 tablet_bootstrap.cc:654] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.482511 1785 raft_consensus.cc:348] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.482715 1785 raft_consensus.cc:374] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.482828 1785 raft_consensus.cc:729] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e59bbbfd1aa4f4898d18898f216ce58, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.483147 1785 consensus_queue.cc:260] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.485165 1785 ts_tablet_manager.cc:1398] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 10:27:40.485426 1785 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58: Bootstrap starting.
I20240430 10:27:40.486964 1785 tablet_bootstrap.cc:654] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.489218 1786 raft_consensus.cc:348] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.489882 1786 raft_consensus.cc:374] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.490108 1786 raft_consensus.cc:729] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e663590b34d4d23889e203c7f2b7d41, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.491393 1786 consensus_queue.cc:260] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.496536 1786 ts_tablet_manager.cc:1398] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41: Time spent starting tablet: real 0.020s user 0.010s sys 0.004s
I20240430 10:27:40.496817 1784 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145: No bootstrap required, opened a new log
I20240430 10:27:40.496977 1784 ts_tablet_manager.cc:1367] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.002s
I20240430 10:27:40.497021 1766 heartbeater.cc:503] Master 127.1.63.62:41335 was elected leader, sending a full tablet report...
I20240430 10:27:40.497851 1784 raft_consensus.cc:348] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.498050 1784 raft_consensus.cc:374] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.498121 1784 raft_consensus.cc:729] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b95ae9cf4a42c08032faef05080145, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.498334 1784 consensus_queue.cc:260] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.498888 1784 ts_tablet_manager.cc:1398] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240430 10:27:40.499159 1784 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: Bootstrap starting.
I20240430 10:27:40.500746 1786 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41: Bootstrap starting.
I20240430 10:27:40.501319 1784 tablet_bootstrap.cc:654] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.506669 1786 tablet_bootstrap.cc:654] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.509757 1785 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58: No bootstrap required, opened a new log
I20240430 10:27:40.509860 1784 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: No bootstrap required, opened a new log
I20240430 10:27:40.509950 1785 ts_tablet_manager.cc:1367] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.013s
I20240430 10:27:40.510008 1786 tablet_bootstrap.cc:492] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41: No bootstrap required, opened a new log
I20240430 10:27:40.510015 1784 ts_tablet_manager.cc:1367] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240430 10:27:40.510138 1786 ts_tablet_manager.cc:1367] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240430 10:27:40.510798 1784 raft_consensus.cc:348] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.510797 1785 raft_consensus.cc:348] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.511065 1785 raft_consensus.cc:374] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.511143 1785 raft_consensus.cc:729] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e59bbbfd1aa4f4898d18898f216ce58, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.511051 1786 raft_consensus.cc:348] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.511341 1786 raft_consensus.cc:374] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.511453 1786 raft_consensus.cc:729] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e663590b34d4d23889e203c7f2b7d41, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.511384 1785 consensus_queue.cc:260] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.511572 1784 raft_consensus.cc:374] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.511771 1786 consensus_queue.cc:260] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.512090 1785 ts_tablet_manager.cc:1398] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240430 10:27:40.511658 1784 raft_consensus.cc:729] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b95ae9cf4a42c08032faef05080145, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.512435 1784 consensus_queue.cc:260] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.512912 1784 ts_tablet_manager.cc:1398] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 10:27:40.513430 1786 ts_tablet_manager.cc:1398] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 10:27:40.513672 1786 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: Bootstrap starting.
I20240430 10:27:40.515328 1786 tablet_bootstrap.cc:654] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: Neither blocks nor log segments found. Creating new log.
I20240430 10:27:40.517274 1786 tablet_bootstrap.cc:492] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: No bootstrap required, opened a new log
I20240430 10:27:40.517447 1786 ts_tablet_manager.cc:1367] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240430 10:27:40.517971 1786 raft_consensus.cc:348] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.518143 1786 raft_consensus.cc:374] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:27:40.518194 1786 raft_consensus.cc:729] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e663590b34d4d23889e203c7f2b7d41, State: Initialized, Role: FOLLOWER
I20240430 10:27:40.518406 1786 consensus_queue.cc:260] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.518811 1786 ts_tablet_manager.cc:1398] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240430 10:27:40.567283 1767 tablet.cc:2369] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 10:27:40.593699 1637 tablet.cc:2369] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:27:40.604665 1790 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:27:40.604952 1790 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.606663 1790 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:27:40.614091 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "45b95ae9cf4a42c08032faef05080145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41" is_pre_election: true
I20240430 10:27:40.614094 1460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "45b95ae9cf4a42c08032faef05080145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" is_pre_election: true
I20240430 10:27:40.614562 1721 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b95ae9cf4a42c08032faef05080145 in term 0.
I20240430 10:27:40.614570 1460 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b95ae9cf4a42c08032faef05080145 in term 0.
I20240430 10:27:40.615121 1526 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 3e663590b34d4d23889e203c7f2b7d41, 45b95ae9cf4a42c08032faef05080145; no voters:
I20240430 10:27:40.615501 1790 raft_consensus.cc:2781] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:27:40.615643 1790 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:27:40.615746 1790 raft_consensus.cc:3037] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.618238 1790 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.618849 1790 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [CANDIDATE]: Term 1 election: Requested vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:27:40.619168 1460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "45b95ae9cf4a42c08032faef05080145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e59bbbfd1aa4f4898d18898f216ce58"
I20240430 10:27:40.619278 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "45b95ae9cf4a42c08032faef05080145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41"
I20240430 10:27:40.619335 1460 raft_consensus.cc:3037] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.619462 1721 raft_consensus.cc:3037] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.621392 1460 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b95ae9cf4a42c08032faef05080145 in term 1.
I20240430 10:27:40.621488 1721 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b95ae9cf4a42c08032faef05080145 in term 1.
I20240430 10:27:40.621759 1524 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145; no voters:
W20240430 10:27:40.621893 1506 tablet.cc:2369] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:27:40.622048 1790 raft_consensus.cc:2781] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:27:40.622516 1790 raft_consensus.cc:686] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [term 1 LEADER]: Becoming Leader. State: Replica: 45b95ae9cf4a42c08032faef05080145, State: Running, Role: LEADER
I20240430 10:27:40.623042 1790 consensus_queue.cc:237] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.627584 1319 catalog_manager.cc:5484] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b95ae9cf4a42c08032faef05080145 (127.1.63.2). New cstate: current_term: 1 leader_uuid: "45b95ae9cf4a42c08032faef05080145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20240430 10:27:40.726634 1795 raft_consensus.cc:482] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:27:40.726843 1795 raft_consensus.cc:504] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.729038 1795 leader_election.cc:288] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
I20240430 10:27:40.734583 1460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a844d0da5c5143bbab129e5a8acee0ec" candidate_uuid: "3e663590b34d4d23889e203c7f2b7d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" is_pre_election: true
I20240430 10:27:40.734869 1460 raft_consensus.cc:2454] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e663590b34d4d23889e203c7f2b7d41 in term 0.
I20240430 10:27:40.735358 1654 leader_election.cc:302] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 3e663590b34d4d23889e203c7f2b7d41; no voters:
I20240430 10:27:40.735538 1591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a844d0da5c5143bbab129e5a8acee0ec" candidate_uuid: "3e663590b34d4d23889e203c7f2b7d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b95ae9cf4a42c08032faef05080145" is_pre_election: true
I20240430 10:27:40.735713 1795 raft_consensus.cc:2781] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:27:40.735829 1795 raft_consensus.cc:482] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:27:40.735917 1795 raft_consensus.cc:3037] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.735890 1591 raft_consensus.cc:2454] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e663590b34d4d23889e203c7f2b7d41 in term 0.
I20240430 10:27:40.738471 1795 raft_consensus.cc:504] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.739084 1795 leader_election.cc:288] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 1 election: Requested vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
I20240430 10:27:40.739527 1460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a844d0da5c5143bbab129e5a8acee0ec" candidate_uuid: "3e663590b34d4d23889e203c7f2b7d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e59bbbfd1aa4f4898d18898f216ce58"
I20240430 10:27:40.739755 1460 raft_consensus.cc:3037] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.739727 1591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a844d0da5c5143bbab129e5a8acee0ec" candidate_uuid: "3e663590b34d4d23889e203c7f2b7d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b95ae9cf4a42c08032faef05080145"
I20240430 10:27:40.739912 1591 raft_consensus.cc:3037] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.742656 1460 raft_consensus.cc:2454] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e663590b34d4d23889e203c7f2b7d41 in term 1.
I20240430 10:27:40.742758 1591 raft_consensus.cc:2454] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e663590b34d4d23889e203c7f2b7d41 in term 1.
I20240430 10:27:40.743113 1654 leader_election.cc:302] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 3e663590b34d4d23889e203c7f2b7d41; no voters:
I20240430 10:27:40.743400 1795 raft_consensus.cc:2781] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:27:40.743764 1795 raft_consensus.cc:686] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [term 1 LEADER]: Becoming Leader. State: Replica: 3e663590b34d4d23889e203c7f2b7d41, State: Running, Role: LEADER
I20240430 10:27:40.744292 1795 consensus_queue.cc:237] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.748646 1319 catalog_manager.cc:5484] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3). New cstate: current_term: 1 leader_uuid: "3e663590b34d4d23889e203c7f2b7d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } health_report { overall_health: HEALTHY } } }
I20240430 10:27:40.850773 1791 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:27:40.850950 1791 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.852664 1791 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:27:40.859659 1591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f07b0d3c8b41bdbd4bef15ff4842f4" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b95ae9cf4a42c08032faef05080145" is_pre_election: true
I20240430 10:27:40.859892 1591 raft_consensus.cc:2454] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 0.
I20240430 10:27:40.860309 1395 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145; no voters:
I20240430 10:27:40.860675 1791 raft_consensus.cc:2781] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:27:40.860772 1791 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:27:40.860869 1791 raft_consensus.cc:3037] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.860939 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f07b0d3c8b41bdbd4bef15ff4842f4" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41" is_pre_election: true
I20240430 10:27:40.861177 1721 raft_consensus.cc:2454] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 0.
I20240430 10:27:40.863066 1791 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.863709 1791 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 1 election: Requested vote from peers 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:27:40.864087 1591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f07b0d3c8b41bdbd4bef15ff4842f4" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b95ae9cf4a42c08032faef05080145"
I20240430 10:27:40.864279 1591 raft_consensus.cc:3037] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.864216 1721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f07b0d3c8b41bdbd4bef15ff4842f4" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41"
I20240430 10:27:40.864424 1721 raft_consensus.cc:3037] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:27:40.866113 1591 raft_consensus.cc:2454] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 1.
I20240430 10:27:40.866210 1721 raft_consensus.cc:2454] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 1.
I20240430 10:27:40.866482 1395 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145; no voters:
I20240430 10:27:40.866734 1791 raft_consensus.cc:2781] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:27:40.867149 1791 raft_consensus.cc:686] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 LEADER]: Becoming Leader. State: Replica: 0e59bbbfd1aa4f4898d18898f216ce58, State: Running, Role: LEADER
I20240430 10:27:40.867607 1791 consensus_queue.cc:237] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:27:40.872749 1319 catalog_manager.cc:5484] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1). New cstate: current_term: 1 leader_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20240430 10:27:40.953203 1276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:27:40.955899 1276 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 0e59bbbfd1aa4f4898d18898f216ce58 to finish bootstrapping
I20240430 10:27:40.964833 1276 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 45b95ae9cf4a42c08032faef05080145 to finish bootstrapping
I20240430 10:27:40.971800 1276 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 3e663590b34d4d23889e203c7f2b7d41 to finish bootstrapping
I20240430 10:27:40.980913 1811 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.1.63.1:34041/threadz?group=all
http://127.1.63.62:37905/masters
http://127.1.63.62:37905/threadz
http://127.1.63.1:34041/mem-trackers
http://127.1.63.3:36649/tablets
http://127.1.63.2:42587/transactions?tablet_id=a844d0da5c5143bbab129e5a8acee0ec
http://127.1.63.62:37905/mem-trackers
http://127.1.63.2:42587/metrics
http://127.1.63.1:34041/stacks
http://127.1.63.3:36649/maintenance-manager
http://127.1.63.2:42587/startup
http://127.1.63.62:37905/ipki-ca-cert-pem
http://127.1.63.2:42587/maintenance-manager
http://127.1.63.1:34041/startup
http://127.1.63.62:37905/threadz?group=all
http://127.1.63.1:34041/tablets
http://127.1.63.3:36649/transactions?tablet_id=a844d0da5c5143bbab129e5a8acee0ec
http://127.1.63.2:42587/stacks
http://127.1.63.2:42587/threadz
http://127.1.63.3:36649/startup
http://127.1.63.3:36649/mem-trackers
http://127.1.63.3:36649/threadz?group=all
http://127.1.63.62:37905/ipki-ca-cert
http://127.1.63.3:36649/threadz
http://127.1.63.62:37905/tables
http://127.1.63.2:42587/mem-trackers
http://127.1.63.3:36649/scans
http://127.1.63.1:34041/maintenance-manager
http://127.1.63.2:42587/tablets
http://127.1.63.1:34041/threadz
http://127.1.63.62:37905/dump-entities
http://127.1.63.62:37905/startup
http://127.1.63.1:34041/transactions?tablet_id=a844d0da5c5143bbab129e5a8acee0ec
http://127.1.63.2:42587/scans
http://127.1.63.2:42587/threadz?group=all
http://127.1.63.62:37905/ipki-ca-cert-der
http://127.1.63.62:37905/tablet-servers
http://127.1.63.62:37905/stacks
http://127.1.63.1:34041/scans
http://127.1.63.3:36649/stacks
http://127.1.63.62:37905/metrics
http://127.1.63.1:34041/metrics
http://127.1.63.3:36649/metrics
I20240430 10:27:40.985703 1276 linked_list-test-util.h:401] Running for: 30.000s
I20240430 10:27:40.985845 1276 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 10:27:40.985956 1276 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240430 10:27:41.036671 1460 raft_consensus.cc:1261] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Refusing update from remote peer 45b95ae9cf4a42c08032faef05080145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.037434 1459 raft_consensus.cc:1261] T a844d0da5c5143bbab129e5a8acee0ec P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Refusing update from remote peer 3e663590b34d4d23889e203c7f2b7d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.037894 1721 raft_consensus.cc:1261] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Refusing update from remote peer 45b95ae9cf4a42c08032faef05080145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.038105 1591 raft_consensus.cc:1261] T a844d0da5c5143bbab129e5a8acee0ec P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Refusing update from remote peer 3e663590b34d4d23889e203c7f2b7d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.038532 1801 consensus_queue.cc:1035] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.038759 1790 consensus_queue.cc:1035] T 612ef01002fd49128ee44f71a75c4690 P 45b95ae9cf4a42c08032faef05080145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.038846 1795 consensus_queue.cc:1035] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.039255 1804 consensus_queue.cc:1035] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.039817 1721 raft_consensus.cc:1261] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Refusing update from remote peer 0e59bbbfd1aa4f4898d18898f216ce58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.040416 1590 raft_consensus.cc:1261] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 45b95ae9cf4a42c08032faef05080145 [term 1 FOLLOWER]: Refusing update from remote peer 0e59bbbfd1aa4f4898d18898f216ce58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:27:41.042517 1808 consensus_queue.cc:1035] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.043183 1808 consensus_queue.cc:1035] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:27:41.068034 1821 mvcc.cc:205] Tried to move back new op lower bound from 7022480838793478144 to 7022480837612515328. Current Snapshot: MvccSnapshot[applied={T|T < 7022480838793478144}]
I20240430 10:27:41.071280 1820 mvcc.cc:205] Tried to move back new op lower bound from 7022480838808543232 to 7022480838119936000. Current Snapshot: MvccSnapshot[applied={T|T < 7022480838808543232}]
I20240430 10:27:41.085722 1819 mvcc.cc:205] Tried to move back new op lower bound from 7022480838793478144 to 7022480837612515328. Current Snapshot: MvccSnapshot[applied={T|T < 7022480838793478144}]
I20240430 10:27:42.873433 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:42.912652 1839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:43.662266 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:46.506596 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:46.532869 1839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:46.582531 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:27:46.608794 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:27:46.640952 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:27:48.415390 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:48.687561 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 2.104s user 0.605s sys 0.080s Metrics: {"bytes_written":3200,"cfile_init":1,"compiler_manager_pool.queue_time_us":432075,"compiler_manager_pool.run_cpu_time_us":677283,"compiler_manager_pool.run_wall_time_us":1703720,"delete_count":0,"dirs.queue_time_us":25184,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":11377,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2283,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8098,"thread_start_us":40543,"threads_started":2,"update_count":471}
I20240430 10:27:48.694073 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690): perf score=0.024951
I20240430 10:27:48.886257 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.192s user 0.075s sys 0.009s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4108,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":17459,"dirs.run_cpu_time_us":355,"dirs.run_wall_time_us":13570,"lbm_read_time_us":559,"lbm_reads_lt_1ms":24,"lbm_write_time_us":512,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":34784,"threads_started":2,"update_count":471}
I20240430 10:27:48.888024 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 140178 bytes on disk
I20240430 10:27:48.888798 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240430 10:27:49.092476 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 2.450s user 0.566s sys 0.130s Metrics: {"bytes_written":3414,"cfile_init":1,"compiler_manager_pool.queue_time_us":317043,"compiler_manager_pool.run_cpu_time_us":675553,"compiler_manager_pool.run_wall_time_us":2228204,"delete_count":0,"dirs.queue_time_us":61065,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":7746,"drs_written":1,"lbm_read_time_us":180,"lbm_reads_lt_1ms":4,"lbm_write_time_us":25402,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8131,"thread_start_us":126871,"threads_started":2,"update_count":504}
I20240430 10:27:49.095189 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 140566 bytes on disk
I20240430 10:27:49.096297 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240430 10:27:49.097141 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690): perf score=0.026509
I20240430 10:27:49.100689 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 2.491s user 0.570s sys 0.100s Metrics: {"bytes_written":3200,"cfile_init":1,"compiler_manager_pool.queue_time_us":318874,"compiler_manager_pool.run_cpu_time_us":655807,"compiler_manager_pool.run_wall_time_us":2379618,"delete_count":0,"dirs.queue_time_us":17939,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":12632,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2317,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8098,"thread_start_us":26367,"threads_started":2,"update_count":471}
I20240430 10:27:49.102793 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690): perf score=0.024951
I20240430 10:27:49.201874 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.104s user 0.045s sys 0.038s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4354,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":7121,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1298,"lbm_read_time_us":503,"lbm_reads_lt_1ms":20,"lbm_write_time_us":310,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":13593,"threads_started":2,"update_count":504}
I20240430 10:27:49.254951 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.134s user 0.074s sys 0.029s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4108,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":15548,"dirs.run_cpu_time_us":295,"dirs.run_wall_time_us":2205,"lbm_read_time_us":521,"lbm_reads_lt_1ms":24,"lbm_write_time_us":501,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":28022,"threads_started":2,"update_count":471}
I20240430 10:27:49.256079 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 140178 bytes on disk
I20240430 10:27:49.257273 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240430 10:27:50.995954 1276 linked_list-test-util.h:417] Sample at HT timestamp: 7022480879538266112 Inserted count: 19700
I20240430 10:27:51.581358 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:51.625560 1839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:52.291574 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:52.902931 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:27:52.962076 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:27:53.020988 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
W20240430 10:27:54.224270 1658 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.413s user 0.001s sys 0.000s
I20240430 10:27:54.228379 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.325s user 0.440s sys 0.106s Metrics: {"bytes_written":145041,"cfile_init":1,"dirs.queue_time_us":6339,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1975,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12306,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":7826,"thread_start_us":6481,"threads_started":1}
I20240430 10:27:54.230120 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014681
I20240430 10:27:54.341066 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.379s user 0.487s sys 0.081s Metrics: {"bytes_written":4379,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6512,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2215,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2051,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7860,"thread_start_us":6653,"threads_started":1,"update_count":653}
I20240430 10:27:54.344472 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014682
I20240430 10:27:54.414912 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.393s user 0.459s sys 0.111s Metrics: {"bytes_written":4809,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6698,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1834,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2008,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7959,"thread_start_us":6833,"threads_started":1,"update_count":719}
I20240430 10:27:54.416919 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 135774 bytes on disk
I20240430 10:27:54.417636 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240430 10:27:54.418560 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014817
I20240430 10:27:55.725329 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.307s user 0.666s sys 0.014s Metrics: {"bytes_written":295275,"cfile_cache_hit":14,"cfile_cache_hit_bytes":95801,"cfile_cache_miss":62,"cfile_cache_miss_bytes":977501,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":29890,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":11418,"drs_written":1,"lbm_read_time_us":2954,"lbm_reads_lt_1ms":98,"lbm_write_time_us":7805,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":49,"num_input_rowsets":2,"peak_mem_usage":527725,"rows_written":16057,"thread_start_us":29786,"threads_started":1}
I20240430 10:27:55.726411 1637 maintenance_manager.cc:419] P 45b95ae9cf4a42c08032faef05080145: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 294606 bytes on disk
I20240430 10:27:55.727293 1528 maintenance_manager.cc:643] P 45b95ae9cf4a42c08032faef05080145: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240430 10:27:55.768218 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.538s user 0.658s sys 0.019s Metrics: {"bytes_written":293172,"cfile_cache_hit":15,"cfile_cache_hit_bytes":187991,"cfile_cache_miss":57,"cfile_cache_miss_bytes":869365,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":24973,"dirs.run_cpu_time_us":1020,"dirs.run_wall_time_us":21901,"drs_written":1,"lbm_read_time_us":2805,"lbm_reads_lt_1ms":93,"lbm_write_time_us":5658,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":50,"mutex_wait_us":2,"num_input_rowsets":2,"peak_mem_usage":512751,"rows_written":15924,"thread_start_us":108304,"threads_started":3}
I20240430 10:27:55.770062 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 292076 bytes on disk
I20240430 10:27:55.776638 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240430 10:27:56.213168 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.850s user 0.630s sys 0.022s Metrics: {"bytes_written":294111,"cfile_cache_hit":14,"cfile_cache_hit_bytes":96255,"cfile_cache_miss":62,"cfile_cache_miss_bytes":972733,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":9908,"dirs.run_cpu_time_us":1535,"dirs.run_wall_time_us":21753,"drs_written":1,"lbm_read_time_us":54383,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":3796,"lbm_writes_lt_1ms":51,"mutex_wait_us":64,"num_input_rowsets":2,"peak_mem_usage":525518,"rows_written":15991,"thread_start_us":17597,"threads_started":4}
I20240430 10:27:56.214205 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 293400 bytes on disk
I20240430 10:27:56.214777 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240430 10:27:56.931725 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:56.948871 1839 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:27:57.629232 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:27:57.901104 1524 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.63.1:39971, user_credentials={real_user=slave}} blocked reactor thread for 47268.1us
F20240430 10:27:58.756037 1839 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fafbd21ddcd google::LogMessage::Fail() at ??:0
@ 0x7fafbd221b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fafbd21d7cc google::LogMessage::Flush() at ??:0
@ 0x7fafbd21ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fafbfe46c80 kudu::operator-() at ??:0
@ 0x7fafccac714a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fafcca915de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fafcca91336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fafcca3e2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fafcf50bd87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fafcf51a93c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fafcf51a7a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fafc8f5002a std::function<>::operator()() at ??:0
@ 0x7fafc8f487e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fafc8f437f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fafc8f41927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fafc8f7f089 handle_request at ??:0
@ 0x7fafc8f8176b worker_thread at ??:0
@ 0x7fafc32926db start_thread at ??:0
@ 0x7fafbc0d071f clone at ??:0
W20240430 10:27:59.063369 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: recv got EOF from 127.1.63.2:34979 (error 108)
W20240430 10:27:59.074643 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.074784 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.082365 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.083652 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.099189 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.100091 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.107985 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.111193 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.130604 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.131249 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.142391 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.144551 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.168577 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.186107 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.187086 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.202172 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.224834 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.230207 1395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:27:59.231465 1395 consensus_peers.cc:485] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 10:27:59.237404 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.242357 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.255084 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:27:59.256326 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 10:27:59.256604 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.284708 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.299125 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.304396 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.321906 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.352160 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.371881 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.376516 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.400084 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.458678 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.459813 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.485137 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.489073 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.550562 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.582024 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.583624 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.616698 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.648672 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.683282 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.688078 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:27:59.690079 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:27:59.724006 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.748596 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.755146 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.758126 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.772019 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.779057 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.794829 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.802281 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.802284 1419 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.811570 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.834964 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.840149 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.848922 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.876250 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.880870 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.899296 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.914968 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.921326 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.936928 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.956598 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.963963 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:27:59.999377 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.001083 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.022315 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.041754 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.071236 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.087162 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:00.093935 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.099782 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.133831 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.136675 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:00.153944 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:00.164700 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.180425 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.195839 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
W20240430 10:28:00.233412 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
I20240430 10:28:00.241294 1991 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 45b95ae9cf4a42c08032faef05080145)
I20240430 10:28:00.241498 1991 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:00.242302 1991 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
I20240430 10:28:00.242908 1458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "3e663590b34d4d23889e203c7f2b7d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 722 } } ignore_live_leader: false dest_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" is_pre_election: true
W20240430 10:28:00.244261 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:00.244376 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:00.244630 1991 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:28:00.248240 1990 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 45b95ae9cf4a42c08032faef05080145)
I20240430 10:28:00.248410 1990 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:00.249186 1990 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:28:00.250106 1720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 2 candidate_status { last_received { term: 1 index: 722 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41" is_pre_election: true
I20240430 10:28:00.250402 1720 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 1.
W20240430 10:28:00.250523 1395 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:00.250849 1395 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 3e663590b34d4d23889e203c7f2b7d41; no voters: 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:00.251096 1990 raft_consensus.cc:2781] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 10:28:00.251192 1990 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 45b95ae9cf4a42c08032faef05080145)
I20240430 10:28:00.251268 1990 raft_consensus.cc:3037] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 1 FOLLOWER]: Advancing to term 2
I20240430 10:28:00.252997 1990 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:00.253502 1990 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 election: Requested vote from peers 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979), 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411)
I20240430 10:28:00.254252 1720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612ef01002fd49128ee44f71a75c4690" candidate_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" candidate_term: 2 candidate_status { last_received { term: 1 index: 722 } } ignore_live_leader: false dest_uuid: "3e663590b34d4d23889e203c7f2b7d41"
I20240430 10:28:00.254455 1720 raft_consensus.cc:3037] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Advancing to term 2
W20240430 10:28:00.254644 1395 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:00.256263 1420 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49646: Illegal state: replica 0e59bbbfd1aa4f4898d18898f216ce58 is not leader of this config: current role FOLLOWER
I20240430 10:28:00.256822 1720 raft_consensus.cc:2454] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e59bbbfd1aa4f4898d18898f216ce58 in term 2.
I20240430 10:28:00.257200 1395 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e59bbbfd1aa4f4898d18898f216ce58, 3e663590b34d4d23889e203c7f2b7d41; no voters: 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:00.257486 1990 raft_consensus.cc:2781] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 10:28:00.257661 1990 raft_consensus.cc:686] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [term 2 LEADER]: Becoming Leader. State: Replica: 0e59bbbfd1aa4f4898d18898f216ce58, State: Running, Role: LEADER
I20240430 10:28:00.257916 1990 consensus_queue.cc:237] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 722, Committed index: 722, Last appended: 1.722, Last appended by leader: 722, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:00.261318 1314 catalog_manager.cc:5484] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 reported cstate change: term changed from 1 to 2, leader changed from 45b95ae9cf4a42c08032faef05080145 (127.1.63.2) to 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1). New cstate: current_term: 2 leader_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } health_report { overall_health: UNKNOWN } } }
W20240430 10:28:00.279657 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:00.287606 1720 raft_consensus.cc:1261] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Refusing update from remote peer 0e59bbbfd1aa4f4898d18898f216ce58: Log matching property violated. Preceding OpId in replica: term: 1 index: 722. Preceding OpId from leader: term: 2 index: 724. (index mismatch)
W20240430 10:28:00.288343 1395 consensus_peers.cc:485] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 10:28:00.288386 1990 consensus_queue.cc:1035] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 723, Last known committed idx: 722, Time since last communication: 0.000s
W20240430 10:28:00.290935 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:00.474146 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:28:00.544466 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:28:00.980252 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.506s user 0.410s sys 0.040s Metrics: {"bytes_written":142684,"cfile_init":1,"dirs.queue_time_us":12055,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":3221,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1906,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7718,"thread_start_us":11937,"threads_started":1}
I20240430 10:28:00.981487 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014709
I20240430 10:28:00.986330 1276 linked_list-test-util.h:417] Sample at HT timestamp: 7022480920478695424 Inserted count: 36250
I20240430 10:28:01.140120 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.595s user 0.429s sys 0.008s Metrics: {"bytes_written":146506,"cfile_init":1,"dirs.queue_time_us":6487,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":9524,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1917,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7921,"thread_start_us":6606,"threads_started":1}
I20240430 10:28:01.141243 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014707
W20240430 10:28:01.779006 1395 consensus_peers.cc:485] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 10:28:01.933431 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 10:28:02.330356 1395 consensus_peers.cc:485] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 10:28:02.355378 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:28:02.557102 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.416s user 1.081s sys 0.077s Metrics: {"bytes_written":2124,"cfile_cache_hit":11,"cfile_cache_hit_bytes":47412,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1568616,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":79630,"dirs.run_cpu_time_us":1245,"dirs.run_wall_time_us":22082,"drs_written":1,"lbm_read_time_us":4147,"lbm_reads_lt_1ms":107,"lbm_write_time_us":4893,"lbm_writes_lt_1ms":67,"num_input_rowsets":2,"peak_mem_usage":783256,"reinsert_count":0,"rows_written":23845,"thread_start_us":98206,"threads_started":3,"update_count":300}
I20240430 10:28:02.560253 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690): perf score=0.005645
I20240430 10:28:02.735972 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.754s user 0.999s sys 0.052s Metrics: {"bytes_written":1257,"cfile_cache_hit":11,"cfile_cache_hit_bytes":46216,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1561574,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":31969,"dirs.run_cpu_time_us":1700,"dirs.run_wall_time_us":15831,"drs_written":1,"lbm_read_time_us":4578,"lbm_reads_lt_1ms":107,"lbm_write_time_us":4570,"lbm_writes_lt_1ms":66,"num_input_rowsets":2,"peak_mem_usage":781127,"reinsert_count":0,"rows_written":23709,"thread_start_us":63262,"threads_started":5,"update_count":164}
I20240430 10:28:02.737418 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690): perf score=0.003362
I20240430 10:28:02.868767 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.131s user 0.103s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1888,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":42,"lbm_read_time_us":412,"lbm_reads_lt_1ms":24,"lbm_write_time_us":332,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":164}
I20240430 10:28:02.869830 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 440827 bytes on disk
I20240430 10:28:02.870339 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240430 10:28:02.889806 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: MajorDeltaCompactionOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.329s user 0.147s sys 0.021s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3189,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":3941,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":23,"lbm_write_time_us":325,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":300}
I20240430 10:28:02.890709 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 442404 bytes on disk
I20240430 10:28:02.893886 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240430 10:28:03.097714 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:04.214020 1395 consensus_peers.cc:485] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 10:28:04.314553 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 10:28:04.315696 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 10:28:04.451021 1395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240430 10:28:04.743878 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:04.854741 1395 consensus_peers.cc:485] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 10:28:04.878737 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(01f07b0d3c8b41bdbd4bef15ff4842f4): perf score=1.000000
I20240430 10:28:04.898542 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(01f07b0d3c8b41bdbd4bef15ff4842f4): perf score=1.000000
I20240430 10:28:05.440150 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:28:05.569100 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(01f07b0d3c8b41bdbd4bef15ff4842f4) complete. Timing: real 0.690s user 0.413s sys 0.053s Metrics: {"bytes_written":141124,"cfile_init":1,"dirs.queue_time_us":32442,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":138870,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2682,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7615,"thread_start_us":32348,"threads_started":1}
I20240430 10:28:05.570019 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(01f07b0d3c8b41bdbd4bef15ff4842f4): 142323 bytes on disk
I20240430 10:28:05.570700 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(01f07b0d3c8b41bdbd4bef15ff4842f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240430 10:28:05.571370 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(a844d0da5c5143bbab129e5a8acee0ec): perf score=1.000000
I20240430 10:28:05.875826 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(01f07b0d3c8b41bdbd4bef15ff4842f4) complete. Timing: real 0.974s user 0.509s sys 0.025s Metrics: {"bytes_written":141124,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":164064,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2247,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7615,"thread_start_us":268,"threads_started":1}
I20240430 10:28:05.876919 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(01f07b0d3c8b41bdbd4bef15ff4842f4): 142323 bytes on disk
I20240430 10:28:05.877542 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(01f07b0d3c8b41bdbd4bef15ff4842f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240430 10:28:05.878310 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(a844d0da5c5143bbab129e5a8acee0ec): perf score=1.000000
I20240430 10:28:06.191674 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(a844d0da5c5143bbab129e5a8acee0ec) complete. Timing: real 0.620s user 0.414s sys 0.013s Metrics: {"bytes_written":142118,"cfile_init":1,"dirs.queue_time_us":356,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1996,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7710,"thread_start_us":246,"threads_started":1}
I20240430 10:28:06.193156 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(a844d0da5c5143bbab129e5a8acee0ec): 145944 bytes on disk
I20240430 10:28:06.195289 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(a844d0da5c5143bbab129e5a8acee0ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240430 10:28:06.196159 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
I20240430 10:28:06.599021 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(a844d0da5c5143bbab129e5a8acee0ec) complete. Timing: real 0.721s user 0.465s sys 0.076s Metrics: {"bytes_written":143901,"cfile_init":1,"dirs.queue_time_us":3772,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":23267,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1982,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7802,"thread_start_us":3650,"threads_started":1}
I20240430 10:28:06.600340 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(a844d0da5c5143bbab129e5a8acee0ec): 146929 bytes on disk
I20240430 10:28:06.601040 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(a844d0da5c5143bbab129e5a8acee0ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240430 10:28:06.601922 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling FlushMRSOp(612ef01002fd49128ee44f71a75c4690): perf score=1.000000
W20240430 10:28:06.766285 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 10:28:06.996291 1395 consensus_peers.cc:485] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 10:28:07.090092 1397 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.410s user 0.003s sys 0.000s
I20240430 10:28:07.093958 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.898s user 0.366s sys 0.028s Metrics: {"bytes_written":144432,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":176804,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1987,"lbm_writes_lt_1ms":37,"mutex_wait_us":77,"peak_mem_usage":0,"rows_written":7849}
I20240430 10:28:07.095773 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014685
I20240430 10:28:07.280071 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: FlushMRSOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.678s user 0.509s sys 0.051s Metrics: {"bytes_written":3950,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6308,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":4065,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2435,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7713,"thread_start_us":6435,"threads_started":1,"update_count":588}
I20240430 10:28:07.281154 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 132801 bytes on disk
I20240430 10:28:07.284291 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240430 10:28:07.284968 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690): perf score=1.014684
W20240430 10:28:07.333168 1395 consensus_peers.cc:485] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 10:28:07.627480 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:28:08.537151 1812 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:28:08.560838 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.465s user 1.176s sys 0.057s Metrics: {"bytes_written":463,"cfile_cache_hit":13,"cfile_cache_hit_bytes":93454,"cfile_cache_miss":91,"cfile_cache_miss_bytes":2053758,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":23820,"dirs.run_cpu_time_us":1529,"dirs.run_wall_time_us":36191,"drs_written":1,"lbm_read_time_us":4783,"lbm_reads_lt_1ms":127,"lbm_write_time_us":5215,"lbm_writes_lt_1ms":80,"mutex_wait_us":225,"num_input_rowsets":2,"peak_mem_usage":1035642,"reinsert_count":0,"rows_written":31558,"spinlock_wait_cycles":2944,"thread_start_us":67773,"threads_started":5,"update_count":43}
I20240430 10:28:08.562245 1506 maintenance_manager.cc:419] P 0e59bbbfd1aa4f4898d18898f216ce58: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 597176 bytes on disk
I20240430 10:28:08.562965 1397 maintenance_manager.cc:643] P 0e59bbbfd1aa4f4898d18898f216ce58: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240430 10:28:09.035893 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: CompactRowSetsOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 1.751s user 1.268s sys 0.052s Metrics: {"bytes_written":463,"cfile_cache_hit":13,"cfile_cache_hit_bytes":46434,"cfile_cache_miss":97,"cfile_cache_miss_bytes":2107560,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":55909,"dirs.run_cpu_time_us":1843,"dirs.run_wall_time_us":19091,"drs_written":1,"lbm_read_time_us":4828,"lbm_reads_lt_1ms":133,"lbm_write_time_us":5430,"lbm_writes_lt_1ms":80,"mutex_wait_us":1679,"num_input_rowsets":2,"peak_mem_usage":1046179,"reinsert_count":0,"rows_written":31558,"thread_start_us":64316,"threads_started":5,"update_count":43}
I20240430 10:28:09.037376 1767 maintenance_manager.cc:419] P 3e663590b34d4d23889e203c7f2b7d41: Scheduling UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690): 597176 bytes on disk
I20240430 10:28:09.038242 1658 maintenance_manager.cc:643] P 3e663590b34d4d23889e203c7f2b7d41: UndoDeltaBlockGCOp(612ef01002fd49128ee44f71a75c4690) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240430 10:28:09.325137 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 10:28:09.508523 1395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 10:28:09.509425 1395 consensus_peers.cc:485] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 10:28:09.769229 1395 consensus_peers.cc:485] T 612ef01002fd49128ee44f71a75c4690 P 0e59bbbfd1aa4f4898d18898f216ce58 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 10:28:09.883603 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 10:28:10.616787 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240430 10:28:10.628481 1812 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fc1bbacddcd google::LogMessage::Fail() at ??:0
@ 0x7fc1bbad1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc1bbacd7cc google::LogMessage::Flush() at ??:0
@ 0x7fc1bbacef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc1be6f6c80 kudu::operator-() at ??:0
@ 0x7fc1cb37714a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fc1cb3415de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fc1cb341336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fc1cb2ee2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fc1cddbbd87 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fc1cddca93c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7fc1cddca7a2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc1c780002a std::function<>::operator()() at ??:0
@ 0x7fc1c77f87e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fc1c77f37f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fc1c77f1927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fc1c782f089 handle_request at ??:0
@ 0x7fc1c783176b worker_thread at ??:0
@ 0x7fc1c1b426db start_thread at ??:0
@ 0x7fc1ba98071f clone at ??:0
W20240430 10:28:10.900676 1769 connection.cc:535] client connection to 127.1.63.1:39971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 10:28:10.900956 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 10:28:10.901222 1654 connection.cc:535] client connection to 127.1.63.1:39971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 10:28:10.902518 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 10:28:10.916996 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.918378 1680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.918373 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.939006 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.946018 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.947290 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.969395 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.978514 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:10.999014 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.016026 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.039562 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.070053 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.097173 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.125213 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.159600 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.192099 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.230921 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.265905 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.314247 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.353953 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.406090 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.451354 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.503747 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.560094 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.611424 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.674584 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.726079 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.797839 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.855095 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.918283 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:11.934046 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:11.993242 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.011291 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 10:28:12.041265 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:12.081358 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:12.121769 2137 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:12.121965 2137 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:12.122619 2137 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:12.124202 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:12.124398 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:12.124504 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:12.124751 2137 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 10:28:12.132407 2137 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:12.132536 2137 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:12.133121 2137 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:12.134325 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:12.134770 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:12.134917 1656 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:12.135187 2137 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:12.135336 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.235169 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.291877 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.394733 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.457877 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.569314 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.631208 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.749445 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.817687 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.933415 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:12.943069 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:13.008057 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:13.068121 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:13.130077 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.205719 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.304224 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 10:28:13.336937 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.413910 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.554322 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.632614 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:13.651618 2137 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:13.651808 2137 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:13.652462 2137 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:13.653853 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:13.654078 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:13.654186 1654 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:13.654417 2137 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:13.781225 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:13.856382 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:13.942473 2137 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:13.942716 2137 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:13.943583 2137 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:13.944769 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:13.945068 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:13.945190 1656 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:13.945466 2137 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:13.950129 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:14.094483 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:14.096791 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:14.174633 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.340957 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.420941 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.594383 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.675750 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.736621 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 10:28:14.773172 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.942811 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:14.951421 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:14.958797 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240430 10:28:15.039462 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:15.104727 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:15.214025 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.315640 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.497516 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.601639 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.792085 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.895382 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:15.965849 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 10:28:15.993464 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:16.096580 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:16.111189 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:16.301923 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:16.303682 2137 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:16.303822 2137 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:16.304435 2137 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:16.306231 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:16.306604 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:16.306730 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:16.306970 2137 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 10:28:16.445369 2137 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:16.445588 2137 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:16.446230 2137 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:16.447710 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:16.447934 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:16.448135 1654 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:16.448457 2137 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:16.514454 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:16.614343 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:16.729676 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:16.941030 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:17.050114 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:17.053432 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:17.115736 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:17.364300 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240430 10:28:17.389210 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:17.496870 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:17.621403 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:17.848773 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:17.972954 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:18.085268 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:18.144234 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:18.205214 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:18.337662 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:18.574152 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:18.579809 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 10:28:18.706869 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:18.955415 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:19.086371 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:19.127445 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:19.147565 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:28:19.164206 2142 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:19.164438 2142 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:19.165416 2142 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:19.166877 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:19.167034 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:19.167198 1656 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:19.167469 2142 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:19.476550 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:19.517391 2142 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:19.517566 2142 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:19.518251 2142 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:19.519524 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:19.519760 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:19.519873 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:19.520120 2142 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:19.605304 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:19.741389 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:19.943296 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 10:28:20.007870 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:20.046372 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240430 10:28:20.143324 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:20.144919 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:20.159978 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:20.421916 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:20.702013 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:20.841595 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:20.928777 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 10:28:21.125989 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:21.161916 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:21.183071 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:21.413704 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:21.562773 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:21.720001 2142 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:21.720233 2142 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:21.720854 2142 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:21.722501 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:21.722885 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:21.723026 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:21.723323 2142 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:21.848134 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:22.004205 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:22.182127 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:22.189504 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:22.297856 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:22.453752 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:22.675223 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240430 10:28:22.752437 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:22.916551 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:23.190761 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:23.226142 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:23.227288 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:23.546219 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:23.551721 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 10:28:23.701411 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:23.868242 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:24.185446 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:24.195386 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:24.235989 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:24.352995 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:24.670516 2142 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:24.670724 2142 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:24.671412 2142 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:24.672756 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:24.673269 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:24.673373 1654 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:24.673599 2142 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:24.682243 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:24.851262 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:25.113619 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240430 10:28:25.183847 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:25.200989 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:25.315824 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 10:28:25.353391 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:25.360661 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:25.696833 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:25.878439 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:26.048843 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 10:28:26.219731 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:26.237605 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:26.394580 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:26.405983 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:26.751149 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:26.937124 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:27.233034 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:27.292901 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:27.475965 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:27.480742 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:27.842444 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:27.869079 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 10:28:28.029961 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:28.243592 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:28.403020 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:28.588950 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:28.591503 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:28.651789 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 10:28:28.763069 2142 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:28.763259 2142 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:28.763921 2142 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:28.765512 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:28.765729 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:28.765852 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:28.766079 2142 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:28.778116 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:29.157135 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:29.241469 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:29.351408 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:29.593266 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:29.686604 2142 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:29.686789 2142 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:29.687453 2142 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:29.689700 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:29.690313 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:29.690465 1654 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:29.690757 2142 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:29.929667 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:30.128815 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:30.210003 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 23 similar messages]
I20240430 10:28:30.243014 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:30.349970 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 10:28:30.518304 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:30.714424 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:30.725953 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:31.118113 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:31.260372 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:31.265642 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 10:28:31.328032 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:31.578596 2142 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:31.578856 2142 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:31.579737 2142 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:31.581231 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:31.581493 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:31.581650 1654 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:31.581884 2142 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:28:31.726768 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:31.735368 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:31.938834 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:32.270133 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:32.345433 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:32.553912 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240430 10:28:32.559911 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:32.765130 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:32.973088 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:33.192983 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:33.272977 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:33.604998 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:33.708472 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240430 10:28:33.820317 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:33.831380 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:34.250185 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:34.291826 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:34.480895 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:34.903811 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:34.920917 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:35.085950 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240430 10:28:35.139755 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:35.269764 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240430 10:28:35.307200 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:35.567720 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:35.804873 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:36.016536 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:36.238752 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:36.301290 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240430 10:28:36.316505 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:36.478636 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:36.918769 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:37.145015 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:37.159009 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:37.365597 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:37.606264 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:37.766680 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 10:28:37.850731 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:38.302505 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:38.317417 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:38.370299 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:38.553100 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:38.817767 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 10:28:38.949658 2142 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:38.949823 2142 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:38.950475 2142 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:38.951699 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:38.952188 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:38.952279 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:38.952484 2142 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:39.011557 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:39.262223 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:39.376300 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:39.488178 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:39.727483 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:39.982209 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:40.364248 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:40.448601 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:40.467303 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240430 10:28:40.503063 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:40.523201 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240430 10:28:40.710489 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:41.186741 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:41.373958 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:41.435133 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 10:28:41.443475 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:41.681042 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:41.931494 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:42.188448 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:42.379607 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:42.688349 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:42.691416 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:42.942775 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:43.377099 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240430 10:28:43.380538 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:43.701743 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:43.704531 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:43.754588 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:43.896564 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240430 10:28:44.217832 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:44.268556 2155 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:44.268781 2155 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:44.270396 2155 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:44.270853 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:44.271276 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:44.271360 1656 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:44.271562 2155 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:28:44.385459 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:44.481451 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:44.786106 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:44.996470 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:45.269493 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:45.383667 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:45.563926 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 10:28:45.783775 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:45.796083 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:45.915863 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 10:28:46.061719 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:46.388190 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:46.582396 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:46.710711 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240430 10:28:46.850739 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:46.860373 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:47.385942 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:47.391057 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:47.670941 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:47.938333 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:48.212589 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:48.400748 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:48.408742 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240430 10:28:48.487649 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:49.034826 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:49.041708 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:49.146951 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240430 10:28:49.318686 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:49.411773 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:49.867727 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:50.148618 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:50.156634 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:50.410288 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:50.713817 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:50.775331 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 10:28:50.997308 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240430 10:28:51.003382 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:51.281337 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:51.456725 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:51.567361 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:51.805064 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240430 10:28:51.858340 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:52.426743 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:52.485276 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:52.721513 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:53.003304 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:53.305095 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:53.482210 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:53.497663 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240430 10:28:53.586413 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:53.877312 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:54.184293 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:54.230993 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240430 10:28:54.289961 2155 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:54.290156 2155 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:54.290742 2155 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:54.292201 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:54.292598 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:28:54.292734 1654 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:54.292999 2155 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 10:28:54.501837 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:54.757098 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:54.779220 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:55.076916 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:55.522855 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:55.650148 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:55.888314 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 10:28:55.948369 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:55.962824 2155 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:28:55.962988 2155 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:28:55.963661 2155 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:28:55.965116 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:55.965490 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:28:55.965646 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:28:55.965909 2155 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:28:55.976238 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:56.186740 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240430 10:28:56.546417 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:56.547873 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:56.881529 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:56.918897 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240430 10:28:57.153975 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:57.459654 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:57.567293 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:57.795837 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:58.382750 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:58.414646 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:28:58.561564 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240430 10:28:58.569360 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:58.724776 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:28:59.318107 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:28:59.574829 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:28:59.627403 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240430 10:28:59.631053 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:28:59.656019 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:00.254200 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:00.594507 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:00.594813 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:00.884150 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:01.078330 1654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 10:29:01.200847 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:01.234040 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240430 10:29:01.542068 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:01.656694 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:01.891913 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:02.058883 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240430 10:29:02.156556 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:02.351509 2155 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:29:02.351670 2155 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:29:02.352355 2155 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:29:02.353806 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:02.354105 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:02.354255 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:29:02.354526 2155 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:29:02.499166 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:02.676482 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:03.120509 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:03.142239 1769 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:03.463670 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:03.685865 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:03.737604 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240430 10:29:04.094611 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:04.105942 2155 raft_consensus.cc:482] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:29:04.106128 2155 raft_consensus.cc:504] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:29:04.106799 2155 leader_election.cc:288] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:29:04.108407 1656 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:29:04.109061 1654 leader_election.cc:334] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
I20240430 10:29:04.109196 1654 leader_election.cc:302] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:29:04.109407 2155 raft_consensus.cc:2726] T 01f07b0d3c8b41bdbd4bef15ff4842f4 P 3e663590b34d4d23889e203c7f2b7d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 10:29:04.150846 1769 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:04.440409 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:04.498960 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240430 10:29:04.697765 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:05.077347 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:05.404506 1771 meta_cache.cc:302] tablet 612ef01002fd49128ee44f71a75c4690: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:29:05.424481 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
I20240430 10:29:05.718827 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:06.067193 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:06.347997 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240430 10:29:06.349107 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240430 10:29:06.415942 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:06.460588 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:06.719915 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:06.771818 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240430 10:29:07.068929 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:07.425081 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:07.465296 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:07.743719 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:08.078086 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:08.437770 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:08.477142 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:08.755856 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:29:08.758074 1656 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Couldn't send request to peer 45b95ae9cf4a42c08032faef05080145. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240430 10:29:09.095582 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:09.372655 1654 consensus_peers.cc:485] T a844d0da5c5143bbab129e5a8acee0ec P 3e663590b34d4d23889e203c7f2b7d41 -> Peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Couldn't send request to peer 0e59bbbfd1aa4f4898d18898f216ce58. Status: Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240430 10:29:09.454989 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:09.497982 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:09.756084 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:29:10.100497 2155 raft_consensus.cc:482] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e59bbbfd1aa4f4898d18898f216ce58)
I20240430 10:29:10.100776 2155 raft_consensus.cc:504] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e59bbbfd1aa4f4898d18898f216ce58" member_type: VOTER last_known_addr { host: "127.1.63.1" port: 39971 } } peers { permanent_uuid: "45b95ae9cf4a42c08032faef05080145" member_type: VOTER last_known_addr { host: "127.1.63.2" port: 34979 } } peers { permanent_uuid: "3e663590b34d4d23889e203c7f2b7d41" member_type: VOTER last_known_addr { host: "127.1.63.3" port: 41411 } }
I20240430 10:29:10.101660 2155 leader_election.cc:288] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971), 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979)
W20240430 10:29:10.102932 1654 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e59bbbfd1aa4f4898d18898f216ce58 (127.1.63.1:39971): Network error: Client connection negotiation failed: client connection to 127.1.63.1:39971: connect: Connection refused (error 111)
W20240430 10:29:10.103260 1656 leader_election.cc:334] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979): Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
I20240430 10:29:10.103380 1656 leader_election.cc:302] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e663590b34d4d23889e203c7f2b7d41; no voters: 0e59bbbfd1aa4f4898d18898f216ce58, 45b95ae9cf4a42c08032faef05080145
I20240430 10:29:10.103669 2155 raft_consensus.cc:2726] T 612ef01002fd49128ee44f71a75c4690 P 3e663590b34d4d23889e203c7f2b7d41 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 10:29:10.122691 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:10.487376 1681 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38326: Illegal state: replica 3e663590b34d4d23889e203c7f2b7d41 is not leader of this config: current role FOLLOWER
W20240430 10:29:10.527654 1771 meta_cache.cc:302] tablet 01f07b0d3c8b41bdbd4bef15ff4842f4: replica 45b95ae9cf4a42c08032faef05080145 (127.1.63.2:34979) has failed: Network error: Client connection negotiation failed: client connection to 127.1.63.2:34979: connect: Connection refused (error 111)
W20240430 10:29:10.728834 1771 batcher.cc:441] Timed out: Failed to write batch of 479 ops to tablet a844d0da5c5143bbab129e5a8acee0ec after 1 attempt(s): Failed to write to server: 3e663590b34d4d23889e203c7f2b7d41 (127.1.63.3:41411): Write RPC to 127.1.63.3:41411 timed out after 60.000s (SENT)
I20240430 10:29:10.773123 1822 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:29:10.810851 1276 client-test-util.cc:49] 34 failed ops. First 10 errors follow
I20240430 10:29:10.811039 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-1095669307870806016, int64 link_to=5583928160677330944, int64 insert_ts=1714472890622907 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811292 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-6128432646330777599, int64 link_to=-360187262713004031, int64 insert_ts=1714472890623020 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811393 1276 client-test-util.cc:55] Op INSERT int64 rand_key=2252784750458830851, int64 link_to=6197622276774166531, int64 insert_ts=1714472890623438 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811457 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-3875788633360236540, int64 link_to=5837575751549583364, int64 insert_ts=1714472890623504 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811540 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-8908551971820208123, int64 link_to=-106821146817462267, int64 insert_ts=1714472890623573 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811617 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-527475312518889465, int64 link_to=6450847655181418503, int64 insert_ts=1714472890623679 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811704 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-6655907958849667064, int64 link_to=6090941867445125128, int64 insert_ts=1714472890623727 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811779 1276 client-test-util.cc:55] Op INSERT int64 rand_key=1725168700451651594, int64 link_to=-5797992666777255926, int64 insert_ts=1714472890623837 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811866 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-3307594638008319989, int64 link_to=6704495246053670923, int64 insert_ts=1714472890623882 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 10:29:10.811936 1276 client-test-util.cc:55] Op INSERT int64 rand_key=-1054950625037778930, int64 link_to=-5544767288370003954, int64 insert_ts=1714472890624033 had status Timed out: Failed to write batch of 34 ops to tablet 612ef01002fd49128ee44f71a75c4690 after 342 attempt(s): Failed to write to server: (no server available): Write(tablet: 612ef01002fd49128ee44f71a75c4690, num_ops: 34, num_attempts: 342) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
F20240430 10:29:10.812027 1276 client-test-util.cc:59] Check failed: _s.ok() Bad status: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
*** Check failure stack trace: ***
*** Aborted at 1714472950 (unix time) try "date -d @1714472950" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000004fc) received by PID 1276 (TID 0x7f9c6ff64d80) from PID 1276; stack trace: ***
@ 0x7f9c77971980 (unknown) at ??:0
@ 0x7f9c70a1cfb7 gsignal at ??:0
@ 0x7f9c70a1e921 abort at ??:0
@ 0x7f9c73745dcd google::LogMessage::Fail() at ??:0
@ 0x7f9c73749b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9c737457cc google::LogMessage::Flush() at ??:0
@ 0x7f9c73746f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c831dbefd kudu::client::LogSessionErrorsAndDie() at ??:0
@ 0x3ed8c2 kudu::client::FlushSessionOrDie() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test-util.h:?
@ 0x3d6588 kudu::LinkedListTester::LoadLinkedList() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test-util.h:?
@ 0x3c8889 kudu::LinkedListTest_TestLoadAndVerify_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:?
@ 0x7f9c755e00ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f9c755d4bdb testing::Test::Run() at ??:0
@ 0x7f9c755d4d9d testing::TestInfo::Run() at ??:0
@ 0x7f9c755d5377 testing::TestSuite::Run() at ??:0
@ 0x7f9c755d577c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f9c755e060d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f9c755d4e63 testing::UnitTest::Run() at ??:0
@ 0x7f9c8a2bf3c2 RUN_ALL_TESTS() at ??:0
@ 0x7f9c8a2bd8d9 main at ??:0
@ 0x7f9c709ffbf7 __libc_start_main at ??:0
@ 0x2ebf5a _start at ??:?