LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 1aa766f8e66f4726b0d2adb9abb3c03f, Rpc address: host: "127.20.101.66" port: 43835 (Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111))
I20240430 09:43:04.303748 20885 webui_checker.cc:88] shutting down CURL thread
I20240430 09:43:04.504853 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 20988
I20240430 09:43:04.506178 21110 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:04.717963 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 20988
W20240430 09:43:04.775417 21267 connection.cc:535] client connection to 127.20.101.65:39983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 09:43:04.775758 20913 connection.cc:535] server connection from 127.20.101.65:57107 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 09:43:04.776018 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 21249
W20240430 09:43:04.776491 21267 consensus_peers.cc:485] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983): Couldn't send request to peer 3e2dc1b121d04379820c7a44f613a47a. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:39983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 09:43:04.776988 21371 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:05.024856 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 21249
I20240430 09:43:05.082767 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 20897
I20240430 09:43:05.083937 20958 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:05.199610 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 20897
2024-04-30T09:43:05Z chronyd exiting
I20240430 09:43:05.240064 20885 test_util.cc:175] -----------------------------------------------
I20240430 09:43:05.240195 20885 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 09:42:27.726213 20885 test_util.cc:268] Using random seed: 807541576
I20240430 09:42:27.737886 20885 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 09:42:27.738000 20885 linked_list-test.cc:101] --------------
I20240430 09:42:27.738075 20885 linked_list-test.cc:102] 50 chains
I20240430 09:42:27.738138 20885 linked_list-test.cc:103] 3 tablets
I20240430 09:42:27.738206 20885 linked_list-test.cc:104] Mutations on
I20240430 09:42:27.738263 20885 linked_list-test.cc:105] --------------
I20240430 09:42:27.738510 20885 ts_itest-base.cc:112] Starting cluster with:
I20240430 09:42:27.738574 20885 ts_itest-base.cc:113] --------------
I20240430 09:42:27.738629 20885 ts_itest-base.cc:114] 3 tablet servers
I20240430 09:42:27.738691 20885 ts_itest-base.cc:115] 3 replicas per TS
I20240430 09:42:27.738767 20885 ts_itest-base.cc:116] --------------
2024-04-30T09:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T09:42:27Z Disabled control of system clock
I20240430 09:42:27.762259 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.101.126:38281
--webserver_interface=127.20.101.126
--webserver_port=0
--builtin_ntp_servers=127.20.101.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.101.126:38281
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 09:42:28.091984 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:42:28.092332 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:42:28.092499 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:42:28.099844 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 09:42:28.099975 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:42:28.100062 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 09:42:28.100144 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 09:42:28.109651 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.101.126:38281
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.101.126:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.20.101.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:42:28.111124 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:42:28.112877 20897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:42:28.124161 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:28.124591 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:28.125169 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:42:28.125921 20897 server_base.cc:1034] running on GCE node
I20240430 09:42:28.126817 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:42:28.128244 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:42:28.129529 20897 hybrid_clock.cc:648] HybridClock initialized: now 1714470148129456 us; error 48 us; skew 500 ppm
I20240430 09:42:28.129861 20897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:42:28.132200 20897 webserver.cc:460] Webserver started at http://127.20.101.126:35051/ using document root <none> and password file <none>
I20240430 09:42:28.132901 20897 fs_manager.cc:359] Metadata directory not provided
I20240430 09:42:28.133034 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:42:28.133343 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:42:28.136019 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/instance:
uuid: "57ab7ca1a18541978f4ef1121216790e"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.136929 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal/instance:
uuid: "57ab7ca1a18541978f4ef1121216790e"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.142153 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240430 09:42:28.146253 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:28.147614 20897 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240430 09:42:28.147778 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
uuid: "57ab7ca1a18541978f4ef1121216790e"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.147966 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:42:28.171159 20897 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:42:28.172015 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:42:28.172313 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:42:28.192626 20897 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.126:38281
I20240430 09:42:28.192680 20962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.126:38281 every 8 connection(s)
I20240430 09:42:28.194025 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
I20240430 09:42:28.197897 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:28.198462 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 20897
I20240430 09:42:28.198700 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal/instance
I20240430 09:42:28.208401 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e: Bootstrap starting.
I20240430 09:42:28.211736 20963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:28.213093 20963 log.cc:824] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e: Log is configured to *not* fsync() on all Append() calls
I20240430 09:42:28.215996 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e: No bootstrap required, opened a new log
I20240430 09:42:28.222865 20963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } }
I20240430 09:42:28.223304 20963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:28.223416 20963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ab7ca1a18541978f4ef1121216790e, State: Initialized, Role: FOLLOWER
I20240430 09:42:28.224066 20963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } }
I20240430 09:42:28.224279 20963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 09:42:28.224400 20963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 09:42:28.224558 20963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:28.226825 20963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } }
I20240430 09:42:28.227257 20963 leader_election.cc:302] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ab7ca1a18541978f4ef1121216790e; no voters: 
I20240430 09:42:28.227787 20963 leader_election.cc:288] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 09:42:28.227910 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:42:28.228586 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [term 1 LEADER]: Becoming Leader. State: Replica: 57ab7ca1a18541978f4ef1121216790e, State: Running, Role: LEADER
I20240430 09:42:28.229152 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } }
I20240430 09:42:28.229705 20963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [sys.catalog]: configured and running, proceeding with master startup.
I20240430 09:42:28.232019 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ab7ca1a18541978f4ef1121216790e. Latest consensus state: current_term: 1 leader_uuid: "57ab7ca1a18541978f4ef1121216790e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } } }
I20240430 09:42:28.232326 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [sys.catalog]: This master's current role is: LEADER
I20240430 09:42:28.232512 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ab7ca1a18541978f4ef1121216790e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ab7ca1a18541978f4ef1121216790e" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 38281 } } }
I20240430 09:42:28.232926 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e [sys.catalog]: This master's current role is: LEADER
I20240430 09:42:28.236269 20973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 09:42:28.243986 20973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 09:42:28.253705 20973 catalog_manager.cc:1347] Generated new cluster ID: 5146f89f1dd3419b951cdbbfa1ef3ec0
I20240430 09:42:28.253854 20973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 09:42:28.270351 20973 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 09:42:28.272076 20973 catalog_manager.cc:1504] Loading token signing keys...
I20240430 09:42:28.283288 20973 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 57ab7ca1a18541978f4ef1121216790e: Generated new TSK 0
I20240430 09:42:28.284379 20973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 09:42:28.301087 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.65:0
--local_ip_for_outbound_sockets=127.20.101.65
--webserver_interface=127.20.101.65
--webserver_port=0
--tserver_master_addrs=127.20.101.126:38281
--builtin_ntp_servers=127.20.101.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 09:42:28.655084 20988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:42:28.655369 20988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:42:28.655532 20988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:42:28.663311 20988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:42:28.663443 20988 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:42:28.663480 20988 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:42:28.663534 20988 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:42:28.663671 20988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.65
I20240430 09:42:28.672525 20988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.101.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.101.126:38281
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.65
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:42:28.673826 20988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:42:28.675556 20988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:42:28.688365 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:28.688931 20993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:28.689287 20994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:42:28.690104 20988 server_base.cc:1034] running on GCE node
I20240430 09:42:28.690732 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:42:28.691640 20988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:42:28.692863 20988 hybrid_clock.cc:648] HybridClock initialized: now 1714470148692818 us; error 43 us; skew 500 ppm
I20240430 09:42:28.693210 20988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:42:28.695847 20988 webserver.cc:460] Webserver started at http://127.20.101.65:40827/ using document root <none> and password file <none>
I20240430 09:42:28.696581 20988 fs_manager.cc:359] Metadata directory not provided
I20240430 09:42:28.696692 20988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:42:28.696995 20988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:42:28.699589 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/instance:
uuid: "3e2dc1b121d04379820c7a44f613a47a"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.700294 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/instance:
uuid: "3e2dc1b121d04379820c7a44f613a47a"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.705111 20988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240430 09:42:28.708261 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:28.709618 20988 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240430 09:42:28.709795 20988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
uuid: "3e2dc1b121d04379820c7a44f613a47a"
format_stamp: "Formatted at 2024-04-30 09:42:28 on dist-test-slave-8dh4"
I20240430 09:42:28.709996 20988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:42:28.735226 20988 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:42:28.736065 20988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:42:28.736413 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:42:28.737280 20988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:42:28.738742 20988 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:42:28.738821 20988 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:28.738894 20988 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:42:28.738943 20988 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:28.774233 20988 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.65:39983
I20240430 09:42:28.774276 21114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.65:39983 every 8 connection(s)
I20240430 09:42:28.775513 20988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
I20240430 09:42:28.780133 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 20988
I20240430 09:42:28.780355 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/instance
I20240430 09:42:28.784322 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.66:0
--local_ip_for_outbound_sockets=127.20.101.66
--webserver_interface=127.20.101.66
--webserver_port=0
--tserver_master_addrs=127.20.101.126:38281
--builtin_ntp_servers=127.20.101.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 09:42:28.790508 21115 heartbeater.cc:347] Connected to a master server at 127.20.101.126:38281
I20240430 09:42:28.790809 21115 heartbeater.cc:464] Registering TS with master...
I20240430 09:42:28.791579 21115 heartbeater.cc:511] Master 127.20.101.126:38281 requested a full tablet report, sending...
I20240430 09:42:28.793727 20928 ts_manager.cc:194] Registered new tserver with Master: 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983)
I20240430 09:42:28.796078 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.65:57107
W20240430 09:42:29.120839 21119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:42:29.121101 21119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:42:29.121227 21119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:42:29.128317 21119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:42:29.128448 21119 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:42:29.128494 21119 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:42:29.128998 21119 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:42:29.129079 21119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.66
I20240430 09:42:29.137835 21119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.20.101.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.101.126:38281
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.66
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:42:29.139096 21119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:42:29.140825 21119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:42:29.153348 21125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:29.153856 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:29.154490 21124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:42:29.155216 21119 server_base.cc:1034] running on GCE node
I20240430 09:42:29.155797 21119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:42:29.156719 21119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:42:29.157960 21119 hybrid_clock.cc:648] HybridClock initialized: now 1714470149157911 us; error 29 us; skew 500 ppm
I20240430 09:42:29.158262 21119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:42:29.160501 21119 webserver.cc:460] Webserver started at http://127.20.101.66:43197/ using document root <none> and password file <none>
I20240430 09:42:29.161229 21119 fs_manager.cc:359] Metadata directory not provided
I20240430 09:42:29.161357 21119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:42:29.162140 21119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:42:29.164604 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/instance:
uuid: "1aa766f8e66f4726b0d2adb9abb3c03f"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.165282 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal/instance:
uuid: "1aa766f8e66f4726b0d2adb9abb3c03f"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.169919 21119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20240430 09:42:29.173223 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.174528 21119 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240430 09:42:29.174692 21119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
uuid: "1aa766f8e66f4726b0d2adb9abb3c03f"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.174875 21119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:42:29.201848 21119 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:42:29.202765 21119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:42:29.203038 21119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:42:29.204043 21119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:42:29.205464 21119 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:42:29.205546 21119 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.205653 21119 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:42:29.205729 21119 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.241047 21119 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.66:43835
I20240430 09:42:29.241166 21245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.66:43835 every 8 connection(s)
I20240430 09:42:29.242341 21119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
I20240430 09:42:29.242866 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 21119
I20240430 09:42:29.243043 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal/instance
I20240430 09:42:29.247248 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.67:0
--local_ip_for_outbound_sockets=127.20.101.67
--webserver_interface=127.20.101.67
--webserver_port=0
--tserver_master_addrs=127.20.101.126:38281
--builtin_ntp_servers=127.20.101.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 09:42:29.257789 21246 heartbeater.cc:347] Connected to a master server at 127.20.101.126:38281
I20240430 09:42:29.258069 21246 heartbeater.cc:464] Registering TS with master...
I20240430 09:42:29.258915 21246 heartbeater.cc:511] Master 127.20.101.126:38281 requested a full tablet report, sending...
I20240430 09:42:29.260313 20928 ts_manager.cc:194] Registered new tserver with Master: 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835)
I20240430 09:42:29.261041 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.66:40785
W20240430 09:42:29.578284 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:42:29.578642 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:42:29.578843 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:42:29.586025 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:42:29.586166 21249 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 09:42:29.586216 21249 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 09:42:29.586272 21249 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 09:42:29.586355 21249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.67
I20240430 09:42:29.595695 21249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.20.101.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.101.126:38281
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.67
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:42:29.597014 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:42:29.598681 21249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:42:29.610977 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:29.611546 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:42:29.612161 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:42:29.612831 21249 server_base.cc:1034] running on GCE node
I20240430 09:42:29.613408 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:42:29.614449 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:42:29.615689 21249 hybrid_clock.cc:648] HybridClock initialized: now 1714470149615648 us; error 30 us; skew 500 ppm
I20240430 09:42:29.616003 21249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:42:29.618300 21249 webserver.cc:460] Webserver started at http://127.20.101.67:42921/ using document root <none> and password file <none>
I20240430 09:42:29.619051 21249 fs_manager.cc:359] Metadata directory not provided
I20240430 09:42:29.619164 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:42:29.619475 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:42:29.622090 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/instance:
uuid: "fc63c0f78ed14575a782aab14a47a7a9"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.622829 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal/instance:
uuid: "fc63c0f78ed14575a782aab14a47a7a9"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.627887 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240430 09:42:29.631397 21263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.632725 21249 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240430 09:42:29.632897 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
uuid: "fc63c0f78ed14575a782aab14a47a7a9"
format_stamp: "Formatted at 2024-04-30 09:42:29 on dist-test-slave-8dh4"
I20240430 09:42:29.633087 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:42:29.651636 21249 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:42:29.652493 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:42:29.652776 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:42:29.653709 21249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:42:29.655157 21249 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:42:29.655252 21249 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.655360 21249 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:42:29.655431 21249 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:42:29.692293 21249 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.67:40877
I20240430 09:42:29.692341 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.67:40877 every 8 connection(s)
I20240430 09:42:29.693567 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
I20240430 09:42:29.696137 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 21249
I20240430 09:42:29.696328 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal/instance
I20240430 09:42:29.705651 21376 heartbeater.cc:347] Connected to a master server at 127.20.101.126:38281
I20240430 09:42:29.705931 21376 heartbeater.cc:464] Registering TS with master...
I20240430 09:42:29.706775 21376 heartbeater.cc:511] Master 127.20.101.126:38281 requested a full tablet report, sending...
I20240430 09:42:29.708043 20928 ts_manager.cc:194] Registered new tserver with Master: fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:29.708820 20928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.67:37981
I20240430 09:42:29.712568 20885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:42:29.733273 20928 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51502:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 09:42:29.739334 20928 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 09:42:29.767078 21050 tablet_service.cc:1467] Processing CreateTablet for tablet 2677af7c7e1e4846b54d331fa674a52a (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:42:29.767947 21048 tablet_service.cc:1467] Processing CreateTablet for tablet da44a9806d424d099f2397eeecb0a158 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:42:29.768350 21049 tablet_service.cc:1467] Processing CreateTablet for tablet c980221eb2ae4752adb92064910d0161 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:42:29.768216 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 2677af7c7e1e4846b54d331fa674a52a (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:42:29.769224 21048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da44a9806d424d099f2397eeecb0a158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.768536 21309 tablet_service.cc:1467] Processing CreateTablet for tablet da44a9806d424d099f2397eeecb0a158 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:42:29.769996 21049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c980221eb2ae4752adb92064910d0161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.770638 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2677af7c7e1e4846b54d331fa674a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.771135 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da44a9806d424d099f2397eeecb0a158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.768664 21310 tablet_service.cc:1467] Processing CreateTablet for tablet c980221eb2ae4752adb92064910d0161 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:42:29.773051 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c980221eb2ae4752adb92064910d0161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.772743 21180 tablet_service.cc:1467] Processing CreateTablet for tablet c980221eb2ae4752adb92064910d0161 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 09:42:29.773772 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2677af7c7e1e4846b54d331fa674a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.774350 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c980221eb2ae4752adb92064910d0161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.775347 21179 tablet_service.cc:1467] Processing CreateTablet for tablet da44a9806d424d099f2397eeecb0a158 (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 09:42:29.775805 21179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da44a9806d424d099f2397eeecb0a158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.776564 21181 tablet_service.cc:1467] Processing CreateTablet for tablet 2677af7c7e1e4846b54d331fa674a52a (DEFAULT_TABLE table=TestTable [id=655256ea6d1a4e80a1c168114e6b4ea7]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 09:42:29.777411 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2677af7c7e1e4846b54d331fa674a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:42:29.798736 21396 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Bootstrap starting.
I20240430 09:42:29.798949 21115 heartbeater.cc:503] Master 127.20.101.126:38281 was elected leader, sending a full tablet report...
I20240430 09:42:29.802769 21396 tablet_bootstrap.cc:654] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.804239 21396 log.cc:824] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Log is configured to *not* fsync() on all Append() calls
I20240430 09:42:29.804710 21394 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: Bootstrap starting.
I20240430 09:42:29.805591 21395 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Bootstrap starting.
I20240430 09:42:29.808777 21395 tablet_bootstrap.cc:654] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.810103 21395 log.cc:824] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Log is configured to *not* fsync() on all Append() calls
I20240430 09:42:29.812700 21394 tablet_bootstrap.cc:654] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.814100 21396 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: No bootstrap required, opened a new log
I20240430 09:42:29.814427 21396 ts_tablet_manager.cc:1367] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.007s
I20240430 09:42:29.814721 21394 log.cc:824] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: Log is configured to *not* fsync() on all Append() calls
I20240430 09:42:29.815563 21395 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: No bootstrap required, opened a new log
I20240430 09:42:29.815881 21395 ts_tablet_manager.cc:1367] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240430 09:42:29.817584 21394 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: No bootstrap required, opened a new log
I20240430 09:42:29.817836 21394 ts_tablet_manager.cc:1367] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.002s
I20240430 09:42:29.821260 21396 raft_consensus.cc:348] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.821599 21396 raft_consensus.cc:374] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.821705 21396 raft_consensus.cc:729] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2dc1b121d04379820c7a44f613a47a, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.822312 21396 consensus_queue.cc:260] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.824060 21396 ts_tablet_manager.cc:1398] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.002s
I20240430 09:42:29.824440 21396 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a: Bootstrap starting.
I20240430 09:42:29.824628 21395 raft_consensus.cc:348] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.825016 21395 raft_consensus.cc:374] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.825117 21395 raft_consensus.cc:729] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa766f8e66f4726b0d2adb9abb3c03f, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.825834 21395 consensus_queue.cc:260] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.826112 21396 tablet_bootstrap.cc:654] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.826992 21394 raft_consensus.cc:348] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.827468 21394 raft_consensus.cc:374] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.827596 21394 raft_consensus.cc:729] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc63c0f78ed14575a782aab14a47a7a9, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.829985 21394 consensus_queue.cc:260] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.831471 21376 heartbeater.cc:503] Master 127.20.101.126:38281 was elected leader, sending a full tablet report...
I20240430 09:42:29.832947 21246 heartbeater.cc:503] Master 127.20.101.126:38281 was elected leader, sending a full tablet report...
I20240430 09:42:29.833905 21395 ts_tablet_manager.cc:1398] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.007s
I20240430 09:42:29.834442 21396 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a: No bootstrap required, opened a new log
I20240430 09:42:29.834596 21396 ts_tablet_manager.cc:1367] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.003s
I20240430 09:42:29.835636 21396 raft_consensus.cc:348] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.835909 21396 raft_consensus.cc:374] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.835994 21396 raft_consensus.cc:729] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2dc1b121d04379820c7a44f613a47a, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.836021 21394 ts_tablet_manager.cc:1398] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.003s
I20240430 09:42:29.836282 21396 consensus_queue.cc:260] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.836431 21394 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Bootstrap starting.
I20240430 09:42:29.837604 21396 ts_tablet_manager.cc:1398] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240430 09:42:29.837852 21396 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a: Bootstrap starting.
I20240430 09:42:29.838176 21395 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f: Bootstrap starting.
I20240430 09:42:29.838313 21394 tablet_bootstrap.cc:654] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.839481 21396 tablet_bootstrap.cc:654] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.839921 21395 tablet_bootstrap.cc:654] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.844977 21396 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a: No bootstrap required, opened a new log
I20240430 09:42:29.845149 21396 ts_tablet_manager.cc:1367] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240430 09:42:29.845294 21395 tablet_bootstrap.cc:492] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f: No bootstrap required, opened a new log
I20240430 09:42:29.845459 21395 ts_tablet_manager.cc:1367] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240430 09:42:29.845867 21396 raft_consensus.cc:348] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.846045 21394 tablet_bootstrap.cc:492] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: No bootstrap required, opened a new log
I20240430 09:42:29.846132 21396 raft_consensus.cc:374] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.846196 21394 ts_tablet_manager.cc:1367] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240430 09:42:29.846210 21396 raft_consensus.cc:729] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2dc1b121d04379820c7a44f613a47a, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.846225 21395 raft_consensus.cc:348] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.846465 21395 raft_consensus.cc:374] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.846562 21395 raft_consensus.cc:729] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa766f8e66f4726b0d2adb9abb3c03f, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.846513 21396 consensus_queue.cc:260] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.846787 21395 consensus_queue.cc:260] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.847059 21396 ts_tablet_manager.cc:1398] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 09:42:29.847307 21395 ts_tablet_manager.cc:1398] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 09:42:29.847571 21395 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f: Bootstrap starting.
I20240430 09:42:29.847780 21394 raft_consensus.cc:348] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.847977 21394 raft_consensus.cc:374] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.848063 21394 raft_consensus.cc:729] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc63c0f78ed14575a782aab14a47a7a9, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.848312 21394 consensus_queue.cc:260] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.849174 21395 tablet_bootstrap.cc:654] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.850116 21394 ts_tablet_manager.cc:1398] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240430 09:42:29.850453 21394 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9: Bootstrap starting.
I20240430 09:42:29.852200 21394 tablet_bootstrap.cc:654] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9: Neither blocks nor log segments found. Creating new log.
I20240430 09:42:29.853448 21395 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f: No bootstrap required, opened a new log
I20240430 09:42:29.853611 21395 ts_tablet_manager.cc:1367] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240430 09:42:29.854295 21395 raft_consensus.cc:348] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.854614 21395 raft_consensus.cc:374] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.854722 21395 raft_consensus.cc:729] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa766f8e66f4726b0d2adb9abb3c03f, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.854864 21394 tablet_bootstrap.cc:492] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9: No bootstrap required, opened a new log
I20240430 09:42:29.855010 21394 ts_tablet_manager.cc:1367] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240430 09:42:29.854965 21395 consensus_queue.cc:260] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.855499 21395 ts_tablet_manager.cc:1398] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 09:42:29.855665 21394 raft_consensus.cc:348] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.855880 21394 raft_consensus.cc:374] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:42:29.855957 21394 raft_consensus.cc:729] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc63c0f78ed14575a782aab14a47a7a9, State: Initialized, Role: FOLLOWER
I20240430 09:42:29.856178 21394 consensus_queue.cc:260] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.856592 21394 ts_tablet_manager.cc:1398] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
W20240430 09:42:29.945173 21377 tablet.cc:2369] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:42:29.946004 21400 raft_consensus.cc:482] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:42:29.946270 21400 raft_consensus.cc:504] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.947950 21400 leader_election.cc:288] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:29.949420 21401 raft_consensus.cc:482] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:42:29.949648 21401 raft_consensus.cc:504] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.951550 21401 leader_election.cc:288] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:29.959801 21201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c980221eb2ae4752adb92064910d0161" candidate_uuid: "3e2dc1b121d04379820c7a44f613a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" is_pre_election: true
I20240430 09:42:29.960767 21201 raft_consensus.cc:2454] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2dc1b121d04379820c7a44f613a47a in term 0.
I20240430 09:42:29.961266 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c980221eb2ae4752adb92064910d0161" candidate_uuid: "3e2dc1b121d04379820c7a44f613a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9" is_pre_election: true
I20240430 09:42:29.961623 21006 leader_election.cc:302] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa766f8e66f4726b0d2adb9abb3c03f, 3e2dc1b121d04379820c7a44f613a47a; no voters: 
I20240430 09:42:29.961796 21331 raft_consensus.cc:2454] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2dc1b121d04379820c7a44f613a47a in term 0.
I20240430 09:42:29.962062 21400 raft_consensus.cc:2781] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:42:29.962210 21400 raft_consensus.cc:482] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:42:29.962302 21400 raft_consensus.cc:3037] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.964429 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a" is_pre_election: true
I20240430 09:42:29.964778 21070 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in term 0.
I20240430 09:42:29.965272 21137 leader_election.cc:302] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa766f8e66f4726b0d2adb9abb3c03f, 3e2dc1b121d04379820c7a44f613a47a; no voters: 
I20240430 09:42:29.965428 21400 raft_consensus.cc:504] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.965587 21401 raft_consensus.cc:2781] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:42:29.965724 21401 raft_consensus.cc:482] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:42:29.965790 21401 raft_consensus.cc:3037] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.966173 21400 leader_election.cc:288] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 1 election: Requested vote from peers 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:29.966598 21201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c980221eb2ae4752adb92064910d0161" candidate_uuid: "3e2dc1b121d04379820c7a44f613a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f"
I20240430 09:42:29.966782 21201 raft_consensus.cc:3037] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.967299 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c980221eb2ae4752adb92064910d0161" candidate_uuid: "3e2dc1b121d04379820c7a44f613a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9"
I20240430 09:42:29.967491 21331 raft_consensus.cc:3037] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.967818 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9" is_pre_election: true
I20240430 09:42:29.968104 21330 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in term 0.
I20240430 09:42:29.968822 21401 raft_consensus.cc:504] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.969766 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a"
I20240430 09:42:29.969940 21070 raft_consensus.cc:3037] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.969915 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9"
I20240430 09:42:29.970079 21330 raft_consensus.cc:3037] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:29.970213 21331 raft_consensus.cc:2454] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2dc1b121d04379820c7a44f613a47a in term 1.
I20240430 09:42:29.969427 21401 leader_election.cc:288] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 election: Requested vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:29.970726 21006 leader_election.cc:302] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9; no voters: 
I20240430 09:42:29.969656 21201 raft_consensus.cc:2454] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2dc1b121d04379820c7a44f613a47a in term 1.
I20240430 09:42:29.971221 21400 raft_consensus.cc:2781] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:42:29.972929 21330 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in term 1.
I20240430 09:42:29.973006 21070 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in term 1.
I20240430 09:42:29.973387 21400 raft_consensus.cc:686] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [term 1 LEADER]: Becoming Leader. State: Replica: 3e2dc1b121d04379820c7a44f613a47a, State: Running, Role: LEADER
I20240430 09:42:29.973387 21137 leader_election.cc:302] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa766f8e66f4726b0d2adb9abb3c03f, 3e2dc1b121d04379820c7a44f613a47a; no voters: 
I20240430 09:42:29.973713 21401 raft_consensus.cc:2781] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:42:29.973945 21400 consensus_queue.cc:237] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.974753 21401 raft_consensus.cc:686] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 LEADER]: Becoming Leader. State: Replica: 1aa766f8e66f4726b0d2adb9abb3c03f, State: Running, Role: LEADER
I20240430 09:42:29.975231 21401 consensus_queue.cc:237] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:29.979790 20927 catalog_manager.cc:5484] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65). New cstate: current_term: 1 leader_uuid: "3e2dc1b121d04379820c7a44f613a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } health_report { overall_health: UNKNOWN } } }
I20240430 09:42:29.980968 20926 catalog_manager.cc:5484] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66). New cstate: current_term: 1 leader_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } health_report { overall_health: UNKNOWN } } }
W20240430 09:42:29.994436 21247 tablet.cc:2369] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 09:42:30.028023 21116 tablet.cc:2369] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:42:30.166428 21403 raft_consensus.cc:482] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:42:30.166640 21403 raft_consensus.cc:504] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:30.168179 21403 leader_election.cc:288] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835)
I20240430 09:42:30.175523 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a" is_pre_election: true
I20240430 09:42:30.175810 21070 raft_consensus.cc:2454] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc63c0f78ed14575a782aab14a47a7a9 in term 0.
I20240430 09:42:30.175899 21201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" is_pre_election: true
I20240430 09:42:30.176031 21401 raft_consensus.cc:482] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:42:30.176249 21401 raft_consensus.cc:504] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:30.176344 21267 leader_election.cc:302] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9; no voters: 
I20240430 09:42:30.176548 21201 raft_consensus.cc:2454] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc63c0f78ed14575a782aab14a47a7a9 in term 0.
I20240430 09:42:30.176738 21403 raft_consensus.cc:2781] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:42:30.176858 21403 raft_consensus.cc:482] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:42:30.176951 21403 raft_consensus.cc:3037] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:30.177217 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a" is_pre_election: true
I20240430 09:42:30.177379 21401 leader_election.cc:288] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:30.177405 21070 raft_consensus.cc:2454] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in term 0.
I20240430 09:42:30.177656 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9" is_pre_election: true
I20240430 09:42:30.177752 21137 leader_election.cc:302] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa766f8e66f4726b0d2adb9abb3c03f, 3e2dc1b121d04379820c7a44f613a47a; no voters: 
I20240430 09:42:30.177996 21401 raft_consensus.cc:2781] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:42:30.178067 21401 raft_consensus.cc:482] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:42:30.178185 21401 raft_consensus.cc:3037] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:30.179556 21403 raft_consensus.cc:504] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:30.179994 21330 raft_consensus.cc:2379] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1aa766f8e66f4726b0d2adb9abb3c03f in current term 1: Already voted for candidate fc63c0f78ed14575a782aab14a47a7a9 in this term.
I20240430 09:42:30.180269 21403 leader_election.cc:288] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835)
I20240430 09:42:30.180373 21401 raft_consensus.cc:504] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:30.180747 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a"
I20240430 09:42:30.180884 21401 leader_election.cc:288] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 election: Requested vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:30.180990 21070 raft_consensus.cc:3037] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:42:30.181356 21201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f"
I20240430 09:42:30.181682 21201 raft_consensus.cc:2379] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fc63c0f78ed14575a782aab14a47a7a9 in current term 1: Already voted for candidate 1aa766f8e66f4726b0d2adb9abb3c03f in this term.
I20240430 09:42:30.182083 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9"
I20240430 09:42:30.182358 21330 raft_consensus.cc:2379] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1aa766f8e66f4726b0d2adb9abb3c03f in current term 1: Already voted for candidate fc63c0f78ed14575a782aab14a47a7a9 in this term.
I20240430 09:42:30.182291 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2677af7c7e1e4846b54d331fa674a52a" candidate_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a"
I20240430 09:42:30.183200 21137 leader_election.cc:302] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1aa766f8e66f4726b0d2adb9abb3c03f; no voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9
I20240430 09:42:30.183526 21401 raft_consensus.cc:2726] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 09:42:30.183977 21070 raft_consensus.cc:2454] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc63c0f78ed14575a782aab14a47a7a9 in term 1.
I20240430 09:42:30.184311 21267 leader_election.cc:302] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9; no voters: 1aa766f8e66f4726b0d2adb9abb3c03f
I20240430 09:42:30.184607 21403 raft_consensus.cc:2781] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:42:30.185017 21403 raft_consensus.cc:686] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [term 1 LEADER]: Becoming Leader. State: Replica: fc63c0f78ed14575a782aab14a47a7a9, State: Running, Role: LEADER
I20240430 09:42:30.185482 21403 consensus_queue.cc:237] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:30.189404 20926 catalog_manager.cc:5484] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67). New cstate: current_term: 1 leader_uuid: "fc63c0f78ed14575a782aab14a47a7a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } health_report { overall_health: HEALTHY } } }
I20240430 09:42:30.220520 20885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:42:30.223783 20885 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 3e2dc1b121d04379820c7a44f613a47a to finish bootstrapping
I20240430 09:42:30.233175 20885 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 1aa766f8e66f4726b0d2adb9abb3c03f to finish bootstrapping
I20240430 09:42:30.240914 20885 ts_itest-base.cc:233] Waiting for 3 tablets on tserver fc63c0f78ed14575a782aab14a47a7a9 to finish bootstrapping
I20240430 09:42:30.250057 21423 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
  http://127.20.101.126:35051/ipki-ca-cert-der
  http://127.20.101.126:35051/threadz?group=all
  http://127.20.101.67:42921/metrics
  http://127.20.101.67:42921/transactions?tablet_id=da44a9806d424d099f2397eeecb0a158
  http://127.20.101.66:43197/startup
  http://127.20.101.126:35051/tablet-servers
  http://127.20.101.67:42921/scans
  http://127.20.101.66:43197/maintenance-manager
  http://127.20.101.126:35051/threadz
  http://127.20.101.66:43197/transactions?tablet_id=da44a9806d424d099f2397eeecb0a158
  http://127.20.101.67:42921/startup
  http://127.20.101.126:35051/mem-trackers
  http://127.20.101.67:42921/tablets
  http://127.20.101.66:43197/scans
  http://127.20.101.66:43197/threadz
  http://127.20.101.65:40827/transactions?tablet_id=da44a9806d424d099f2397eeecb0a158
  http://127.20.101.126:35051/stacks
  http://127.20.101.66:43197/threadz?group=all
  http://127.20.101.65:40827/scans
  http://127.20.101.65:40827/startup
  http://127.20.101.65:40827/mem-trackers
  http://127.20.101.65:40827/threadz
  http://127.20.101.67:42921/stacks
  http://127.20.101.126:35051/dump-entities
  http://127.20.101.126:35051/ipki-ca-cert-pem
  http://127.20.101.65:40827/threadz?group=all
  http://127.20.101.126:35051/tables
  http://127.20.101.65:40827/metrics
  http://127.20.101.67:42921/maintenance-manager
  http://127.20.101.126:35051/startup
  http://127.20.101.66:43197/stacks
  http://127.20.101.126:35051/metrics
  http://127.20.101.66:43197/tablets
  http://127.20.101.66:43197/metrics
  http://127.20.101.65:40827/maintenance-manager
  http://127.20.101.67:42921/threadz
  http://127.20.101.67:42921/threadz?group=all
  http://127.20.101.126:35051/ipki-ca-cert
  http://127.20.101.126:35051/masters
  http://127.20.101.66:43197/mem-trackers
  http://127.20.101.67:42921/mem-trackers
  http://127.20.101.65:40827/tablets
  http://127.20.101.65:40827/stacks
I20240430 09:42:30.255506 20885 linked_list-test-util.h:401] Running for: 30.000s
I20240430 09:42:30.255698 20885 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 09:42:30.255832 20885 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240430 09:42:30.265667 21424 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/21390/stat: No such file or directory (error 2)
I20240430 09:42:30.293748 21330 raft_consensus.cc:1261] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Refusing update from remote peer 1aa766f8e66f4726b0d2adb9abb3c03f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.294790 21401 consensus_queue.cc:1035] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.298198 21331 raft_consensus.cc:1261] T c980221eb2ae4752adb92064910d0161 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Refusing update from remote peer 3e2dc1b121d04379820c7a44f613a47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.299036 21201 raft_consensus.cc:1261] T 2677af7c7e1e4846b54d331fa674a52a P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Refusing update from remote peer fc63c0f78ed14575a782aab14a47a7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.299541 21070 raft_consensus.cc:1261] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Refusing update from remote peer fc63c0f78ed14575a782aab14a47a7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.299755 21069 raft_consensus.cc:1261] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Refusing update from remote peer 1aa766f8e66f4726b0d2adb9abb3c03f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.299979 21400 consensus_queue.cc:1035] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.300441 21403 consensus_queue.cc:1035] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.301162 21403 consensus_queue.cc:1035] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.299041 21200 raft_consensus.cc:1261] T c980221eb2ae4752adb92064910d0161 P 1aa766f8e66f4726b0d2adb9abb3c03f [term 1 FOLLOWER]: Refusing update from remote peer 3e2dc1b121d04379820c7a44f613a47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 09:42:30.303301 21415 consensus_queue.cc:1035] T da44a9806d424d099f2397eeecb0a158 P 1aa766f8e66f4726b0d2adb9abb3c03f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.306123 21400 consensus_queue.cc:1035] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:42:30.310768 21432 mvcc.cc:205] Tried to move back new op lower bound from 7022469735589953536 to 7022469734302830592. Current Snapshot: MvccSnapshot[applied={T|T < 7022469735589953536}]
I20240430 09:42:30.349948 21433 mvcc.cc:205] Tried to move back new op lower bound from 7022469735611228160 to 7022469734298714112. Current Snapshot: MvccSnapshot[applied={T|T < 7022469735611228160}]
I20240430 09:42:30.352356 21431 mvcc.cc:205] Tried to move back new op lower bound from 7022469735611228160 to 7022469734298714112. Current Snapshot: MvccSnapshot[applied={T|T < 7022469735611228160}]
I20240430 09:42:30.394523 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:30.416574 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:31.127022 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:33.758342 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:33.793789 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:34.456326 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:35.708319 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:35.760183 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:35.796840 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
W20240430 09:42:38.053397 21007 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.495s	user 0.004s	sys 0.001s
I20240430 09:42:38.169809 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.409s	user 0.554s	sys 0.114s Metrics: {"bytes_written":1630,"cfile_init":1,"compiler_manager_pool.queue_time_us":298266,"compiler_manager_pool.run_cpu_time_us":676865,"compiler_manager_pool.run_wall_time_us":2241189,"delete_count":0,"dirs.queue_time_us":8171,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":33390,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2336,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7962,"thread_start_us":83555,"threads_started":2,"update_count":229}
I20240430 09:42:38.171855 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a): perf score=0.012943
I20240430 09:42:38.181490 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.384s	user 0.526s	sys 0.114s Metrics: {"bytes_written":147127,"cfile_init":1,"compiler_manager_pool.queue_time_us":13429,"dirs.queue_time_us":16054,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":2619,"drs_written":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1847,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7996,"thread_start_us":34604,"threads_started":2}
I20240430 09:42:38.189638 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 140284 bytes on disk
I20240430 09:42:38.190826 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240430 09:42:38.268253 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.558s	user 0.518s	sys 0.135s Metrics: {"bytes_written":1418,"cfile_init":1,"compiler_manager_pool.queue_time_us":474,"delete_count":0,"dirs.queue_time_us":34738,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":180221,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2221,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7927,"thread_start_us":30280,"threads_started":2,"update_count":194}
I20240430 09:42:38.270040 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a): perf score=0.011320
I20240430 09:42:38.395532 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.223s	user 0.070s	sys 0.012s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2242,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":27721,"lbm_read_time_us":578,"lbm_reads_lt_1ms":24,"lbm_write_time_us":360,"lbm_writes_lt_1ms":8,"mutex_wait_us":28,"reinsert_count":0,"spinlock_wait_cycles":21760,"thread_start_us":27433,"threads_started":2,"update_count":229}
I20240430 09:42:38.396535 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 140010 bytes on disk
I20240430 09:42:38.397384 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240430 09:42:38.489976 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.220s	user 0.045s	sys 0.039s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1971,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":21261,"lbm_read_time_us":514,"lbm_reads_lt_1ms":24,"lbm_write_time_us":477,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":21054,"threads_started":1,"update_count":194}
I20240430 09:42:38.491688 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 139604 bytes on disk
I20240430 09:42:38.495076 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240430 09:42:39.235838 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:39.275087 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:39.963456 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:40.256327 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022469776329576448 Inserted count: 19050
I20240430 09:42:42.505156 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:42.659613 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:42.710852 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:42.913249 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:42.969986 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:43.761548 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.256s	user 0.550s	sys 0.050s Metrics: {"bytes_written":7490,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":7456,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":13463,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17488,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7929,"thread_start_us":7616,"threads_started":1,"update_count":1142}
I20240430 09:42:43.772605 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014688
I20240430 09:42:43.948992 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.238s	user 0.503s	sys 0.086s Metrics: {"bytes_written":8771,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":30508,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":5813,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2417,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8061,"spinlock_wait_cycles":1280,"thread_start_us":30751,"threads_started":1,"update_count":1343}
I20240430 09:42:43.954280 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014818
I20240430 09:42:44.304292 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:44.516144 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.856s	user 0.481s	sys 0.113s Metrics: {"bytes_written":8566,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6500,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":5842,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2236,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8064,"thread_start_us":6651,"threads_started":1,"update_count":1312}
I20240430 09:42:44.518944 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 133119 bytes on disk
I20240430 09:42:44.523710 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240430 09:42:44.537902 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014818
I20240430 09:42:45.172036 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.215s	user 0.670s	sys 0.037s Metrics: {"bytes_written":292714,"cfile_cache_hit":13,"cfile_cache_hit_bytes":94711,"cfile_cache_miss":59,"cfile_cache_miss_bytes":977913,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":22172,"dirs.run_cpu_time_us":1019,"dirs.run_wall_time_us":5475,"drs_written":1,"lbm_read_time_us":3361,"lbm_reads_lt_1ms":95,"lbm_write_time_us":3475,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":522995,"rows_written":16057,"spinlock_wait_cycles":5888,"thread_start_us":33662,"threads_started":3}
I20240430 09:42:45.173767 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 295607 bytes on disk
I20240430 09:42:45.175740 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240430 09:42:45.299672 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.526s	user 0.691s	sys 0.031s Metrics: {"bytes_written":289169,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93933,"cfile_cache_miss":62,"cfile_cache_miss_bytes":967639,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":17658,"dirs.run_cpu_time_us":978,"dirs.run_wall_time_us":6199,"drs_written":1,"lbm_read_time_us":3047,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3402,"lbm_writes_lt_1ms":51,"mutex_wait_us":22,"num_input_rowsets":2,"peak_mem_usage":520989,"rows_written":15856,"spinlock_wait_cycles":4992,"thread_start_us":61624,"threads_started":3}
I20240430 09:42:45.300892 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 291510 bytes on disk
I20240430 09:42:45.301431 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240430 09:42:45.441695 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.903s	user 0.751s	sys 0.034s Metrics: {"bytes_written":292070,"cfile_cache_hit":14,"cfile_cache_hit_bytes":94384,"cfile_cache_miss":62,"cfile_cache_miss_bytes":978342,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":12512,"dirs.run_cpu_time_us":1060,"dirs.run_wall_time_us":14156,"drs_written":1,"lbm_read_time_us":3081,"lbm_reads_lt_1ms":98,"lbm_write_time_us":3870,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":526670,"rows_written":16026,"thread_start_us":19113,"threads_started":3}
I20240430 09:42:45.442966 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 294953 bytes on disk
I20240430 09:42:45.443609 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240430 09:42:47.761783 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:47.800860 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:48.498302 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:49.314785 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:49.454063 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:49.687887 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:50.287178 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022469817421447168 Inserted count: 36800
I20240430 09:42:50.544037 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.229s	user 0.531s	sys 0.027s Metrics: {"bytes_written":2331,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9419,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":2448,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6219,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7689,"thread_start_us":9570,"threads_started":1,"update_count":335}
I20240430 09:42:50.546087 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014711
I20240430 09:42:50.863122 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.409s	user 0.456s	sys 0.017s Metrics: {"bytes_written":3377,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6772,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":5421,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12327,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7683,"thread_start_us":6911,"threads_started":1,"update_count":499}
I20240430 09:42:50.864384 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014709
I20240430 09:42:51.124439 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.436s	user 0.454s	sys 0.090s Metrics: {"bytes_written":5337,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":18359,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1160,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2273,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7955,"thread_start_us":18251,"threads_started":1,"update_count":802}
I20240430 09:42:51.128690 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 135611 bytes on disk
I20240430 09:42:51.129453 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240430 09:42:51.131844 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014705
I20240430 09:42:52.574539 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.028s	user 1.017s	sys 0.036s Metrics: {"bytes_written":425246,"cfile_cache_hit":12,"cfile_cache_hit_bytes":92085,"cfile_cache_miss":78,"cfile_cache_miss_bytes":1500873,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":8997,"dirs.run_cpu_time_us":1008,"dirs.run_wall_time_us":16998,"drs_written":1,"lbm_read_time_us":4203,"lbm_reads_lt_1ms":114,"lbm_write_time_us":21911,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":61,"mutex_wait_us":356,"num_input_rowsets":2,"peak_mem_usage":778565,"rows_written":23545,"spinlock_wait_cycles":1664,"thread_start_us":30775,"threads_started":3}
I20240430 09:42:52.575865 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 440808 bytes on disk
I20240430 09:42:52.584714 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.008s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7904,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240430 09:42:52.939620 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:52.995940 21442 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:53.467742 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.336s	user 0.980s	sys 0.055s Metrics: {"bytes_written":436642,"cfile_cache_hit":13,"cfile_cache_hit_bytes":185694,"cfile_cache_miss":77,"cfile_cache_miss_bytes":1439486,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":12022,"dirs.run_cpu_time_us":978,"dirs.run_wall_time_us":15897,"drs_written":1,"lbm_read_time_us":3834,"lbm_reads_lt_1ms":109,"lbm_write_time_us":27261,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":63,"mutex_wait_us":34,"num_input_rowsets":2,"peak_mem_usage":794210,"rows_written":24012,"thread_start_us":59282,"threads_started":3}
I20240430 09:42:53.468992 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 450161 bytes on disk
I20240430 09:42:53.469728 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240430 09:42:53.758731 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 2.879s	user 0.984s	sys 0.077s Metrics: {"bytes_written":427931,"cfile_cache_hit":12,"cfile_cache_hit_bytes":93017,"cfile_cache_miss":78,"cfile_cache_miss_bytes":1512183,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":8942,"dirs.run_cpu_time_us":966,"dirs.run_wall_time_us":16500,"drs_written":1,"lbm_read_time_us":79828,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":113,"lbm_write_time_us":13562,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":62,"mutex_wait_us":36,"num_input_rowsets":2,"peak_mem_usage":784062,"rows_written":23709,"spinlock_wait_cycles":4480,"thread_start_us":62019,"threads_started":3}
I20240430 09:42:53.760004 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 443955 bytes on disk
I20240430 09:42:53.763319 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240430 09:42:53.902967 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:42:55.093061 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(c980221eb2ae4752adb92064910d0161): perf score=1.000000
I20240430 09:42:55.224712 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(c980221eb2ae4752adb92064910d0161): perf score=1.000000
I20240430 09:42:55.269217 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(c980221eb2ae4752adb92064910d0161): perf score=1.000000
I20240430 09:42:55.970366 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.877s	user 0.403s	sys 0.039s Metrics: {"bytes_written":1861,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":7729,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":11664,"drs_written":1,"lbm_read_time_us":576,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2170,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7626,"spinlock_wait_cycles":5504,"thread_start_us":7851,"threads_started":1,"update_count":263}
I20240430 09:42:55.978067 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161): 142260 bytes on disk
I20240430 09:42:55.978798 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240430 09:42:55.979534 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(da44a9806d424d099f2397eeecb0a158): perf score=1.000000
I20240430 09:42:55.982780 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: FlushMRSOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.713s	user 0.493s	sys 0.023s Metrics: {"bytes_written":2167,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6525,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":9640,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2212,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7673,"thread_start_us":6649,"threads_started":1,"update_count":310}
I20240430 09:42:55.993121 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161): 142750 bytes on disk
I20240430 09:42:55.999341 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240430 09:42:56.000344 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(da44a9806d424d099f2397eeecb0a158): perf score=1.000000
I20240430 09:42:56.986881 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 1.762s	user 0.445s	sys 0.030s Metrics: {"bytes_written":2095,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13870,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":6818,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2758,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7663,"thread_start_us":14027,"threads_started":1,"update_count":300}
I20240430 09:42:57.003134 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 1.023s	user 0.446s	sys 0.055s Metrics: {"bytes_written":978,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":10996,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":2877,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2244,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7738,"thread_start_us":10893,"threads_started":1,"update_count":125}
I20240430 09:42:57.005321 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:57.007156 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161): 142664 bytes on disk
I20240430 09:42:57.014914 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240430 09:42:57.016737 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:57.549865 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: FlushMRSOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 1.543s	user 0.475s	sys 0.047s Metrics: {"bytes_written":978,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11717,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":4962,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":27855,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7738,"thread_start_us":11572,"threads_started":1,"update_count":125}
I20240430 09:42:57.551013 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158): 145637 bytes on disk
I20240430 09:42:57.551802 21138 maintenance_manager.cc:643] P 1aa766f8e66f4726b0d2adb9abb3c03f: UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240430 09:42:57.552655 21247 maintenance_manager.cc:419] P 1aa766f8e66f4726b0d2adb9abb3c03f: Scheduling FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.000000
I20240430 09:42:57.696386 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.685s	user 0.474s	sys 0.025s Metrics: {"bytes_written":161475,"cfile_init":1,"dirs.queue_time_us":6461,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":5789,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2077,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":8632,"thread_start_us":6596,"threads_started":1}
I20240430 09:42:57.697615 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling LogGCOp(2677af7c7e1e4846b54d331fa674a52a): free 1049408 bytes of WAL
I20240430 09:42:57.698231 21268 log_reader.cc:386] T 2677af7c7e1e4846b54d331fa674a52a: removed 1 log segments from log reader
I20240430 09:42:57.698426 21268 log.cc:1076] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9: Deleting log segment in path: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal/wals/2677af7c7e1e4846b54d331fa674a52a/wal-000000001 (ops 1-738)
I20240430 09:42:57.699151 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: LogGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240430 09:42:57.699761 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014678
I20240430 09:42:57.903723 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240430 09:42:57.913877 21442 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7fe0a07addcd  google::LogMessage::Fail() at ??:0
    @     0x7fe0a07b1b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fe0a07ad7cc  google::LogMessage::Flush() at ??:0
    @     0x7fe0a07aef59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fe0a33d6c80  kudu::operator-() at ??:0
    @     0x7fe0b005714a  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7fe0b00215de  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7fe0b0021336  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7fe0affce2d9  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7fe0b2a9bd87  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7fe0b2aaa93c  kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7fe0b2aaa7a2  std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7fe0ac4e002a  std::function<>::operator()() at ??:0
    @     0x7fe0ac4d87e6  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7fe0ac4d37f1  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7fe0ac4d1927  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7fe0ac50f089  handle_request at ??:0
    @     0x7fe0ac51176b  worker_thread at ??:0
    @     0x7fe0a68226db  start_thread at ??:0
    @     0x7fe09f66071f  clone at ??:0
W20240430 09:42:58.447883 21007 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.441s	user 0.000s	sys 0.000s
I20240430 09:42:58.452726 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.436s	user 0.448s	sys 0.077s Metrics: {"bytes_written":149583,"cfile_init":1,"dirs.queue_time_us":62935,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":12925,"drs_written":1,"lbm_read_time_us":1326,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2366,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8132,"thread_start_us":63226,"threads_started":1}
I20240430 09:42:58.454183 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling LogGCOp(2677af7c7e1e4846b54d331fa674a52a): free 1049054 bytes of WAL
I20240430 09:42:58.454603 21007 log_reader.cc:386] T 2677af7c7e1e4846b54d331fa674a52a: removed 1 log segments from log reader
I20240430 09:42:58.454722 21007 log.cc:1076] T 2677af7c7e1e4846b54d331fa674a52a P 3e2dc1b121d04379820c7a44f613a47a: Deleting log segment in path: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/wals/2677af7c7e1e4846b54d331fa674a52a/wal-000000001 (ops 1-739)
I20240430 09:42:58.455391 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: LogGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240430 09:42:58.455938 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 141177 bytes on disk
I20240430 09:42:58.456545 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240430 09:42:58.460451 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a): perf score=1.014574
W20240430 09:42:58.502003 21382 meta_cache.cc:302] tablet da44a9806d424d099f2397eeecb0a158: replica 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835) has failed: Network error: recv got EOF from 127.20.101.66:43835 (error 108)
W20240430 09:42:58.503376 21267 connection.cc:535] client connection to 127.20.101.66:43835 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 09:42:58.503633 21267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 09:42:58.505813 21267 consensus_peers.cc:485] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 09:42:58.509649 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.517573 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.547825 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.552917 21006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111)
W20240430 09:42:58.555163 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.565251 21006 consensus_peers.cc:485] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 09:42:58.578094 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.590631 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.618168 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.632627 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.669894 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.687623 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.707366 21290 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.715225 21030 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.729537 21030 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.733687 21030 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.742087 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.753417 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
I20240430 09:42:58.771003 21605 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40166 (ReqId={client: cafb7dac60aa4d55bf8cc7ad1571dd50, seq_no=2988, attempt_no=0}) took 1127 ms. Trace:
W20240430 09:42:58.771590 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
I20240430 09:42:58.771203 21605 rpcz_store.cc:276] 0430 09:42:57.643163 (+     0us) service_pool.cc:168] Inserting onto call queue
0430 09:42:57.643224 (+    61us) service_pool.cc:225] Handling call
0430 09:42:58.770979 (+1127755us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 09:42:57.644341 (+     0us) write_op.cc:183] PREPARE: starting on tablet 2677af7c7e1e4846b54d331fa674a52a
0430 09:42:57.644454 (+   113us) write_op.cc:424] Acquiring schema lock in shared mode
0430 09:42:57.644459 (+     5us) write_op.cc:427] Acquired schema lock
0430 09:42:57.644461 (+     2us) tablet.cc:660] Decoding operations
0430 09:42:57.653280 (+  8819us) write_op.cc:612] Acquiring the partition lock for write op
0430 09:42:57.653296 (+    16us) write_op.cc:633] Partition lock acquired for write op
0430 09:42:57.653302 (+     6us) tablet.cc:683] Acquiring locks for 1083 operations
0430 09:42:57.704045 (+ 50743us) tablet.cc:699] Row locks acquired
0430 09:42:57.704049 (+     4us) write_op.cc:260] PREPARE: finished
0430 09:42:57.704113 (+    64us) write_op.cc:270] Start()
0430 09:42:57.704151 (+    38us) write_op.cc:276] Timestamp: P: 1714470177704077 usec, L: 0
0430 09:42:57.704156 (+     5us) op_driver.cc:331] REPLICATION: starting
0430 09:42:57.704432 (+   276us) log.cc:842] Serialized 12258 byte log entry
0430 09:42:57.715673 (+ 11241us) op_driver.cc:444] REPLICATION: finished
0430 09:42:57.715931 (+   258us) write_op.cc:301] APPLY: starting
0430 09:42:57.715949 (+    18us) tablet.cc:1365] starting BulkCheckPresence
0430 09:42:58.330892 (+614943us) tablet.cc:1368] finished BulkCheckPresence
0430 09:42:58.330939 (+    47us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 09:42:58.765745 (+434806us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 09:42:58.766785 (+  1040us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=2166,key_file_lookups=1058,delta_file_lookups=0,mrs_lookups=554
0430 09:42:58.766809 (+    24us) write_op.cc:312] APPLY: finished
0430 09:42:58.769510 (+  2701us) log.cc:842] Serialized 7576 byte log entry
0430 09:42:58.769612 (+   102us) write_op.cc:481] Releasing partition, row and schema locks
0430 09:42:58.770768 (+  1156us) write_op.cc:446] Released schema lock
0430 09:42:58.770868 (+   100us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":181,"cfile_cache_hit":2553,"cfile_cache_hit_bytes":55307874,"cfile_cache_miss":4,"cfile_cache_miss_bytes":54806,"cfile_init":1,"lbm_read_time_us":198,"lbm_reads_lt_1ms":8,"mutex_wait_us":598,"num_ops":1083,"prepare.queue_time_us":648,"prepare.run_cpu_time_us":16042,"prepare.run_wall_time_us":122243,"raft.queue_time_us":1986,"raft.run_cpu_time_us":840,"raft.run_wall_time_us":840,"replication_time_us":11489,"spinlock_wait_cycles":1408}]]}
W20240430 09:42:58.791154 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.799408 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.821626 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.823511 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.839813 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.873508 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.880903 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.893606 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.907114 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.935678 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:58.957763 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:58.970746 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.003809 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.006151 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.031437 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.071949 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.089414 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.108161 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.117579 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.178256 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
I20240430 09:42:59.194846 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:42:59.203665 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.218606 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.244029 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
I20240430 09:42:59.264724 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.565s	user 1.279s	sys 0.024s Metrics: {"bytes_written":3595,"cfile_cache_hit":11,"cfile_cache_hit_bytes":51573,"cfile_cache_miss":91,"cfile_cache_miss_bytes":2138317,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":3202,"dirs.run_cpu_time_us":756,"dirs.run_wall_time_us":3863,"drs_written":1,"lbm_read_time_us":5659,"lbm_reads_lt_1ms":123,"lbm_write_time_us":7040,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":79,"mutex_wait_us":44,"num_input_rowsets":2,"peak_mem_usage":1058219,"reinsert_count":0,"rows_written":32177,"thread_start_us":3099,"threads_started":8,"update_count":529}
I20240430 09:42:59.265822 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling MajorDeltaCompactionOp(c980221eb2ae4752adb92064910d0161): perf score=0.015526
W20240430 09:42:59.297421 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
I20240430 09:42:59.306406 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: MajorDeltaCompactionOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.040s	user 0.032s	sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2522,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":88,"lbm_read_time_us":374,"lbm_reads_lt_1ms":20,"lbm_write_time_us":307,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":263}
I20240430 09:42:59.307204 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling MajorDeltaCompactionOp(da44a9806d424d099f2397eeecb0a158): perf score=0.008058
W20240430 09:42:59.312235 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.339234 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
W20240430 09:42:59.347530 21291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40166: Illegal state: replica fc63c0f78ed14575a782aab14a47a7a9 is not leader of this config: current role FOLLOWER
I20240430 09:42:59.349329 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: MajorDeltaCompactionOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.042s	user 0.039s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1389,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":180,"lbm_read_time_us":441,"lbm_reads_lt_1ms":24,"lbm_write_time_us":360,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":274,"threads_started":1,"update_count":125,"wal-append.queue_time_us":416}
I20240430 09:42:59.350441 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 605392 bytes on disk
I20240430 09:42:59.351047 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240430 09:42:59.351758 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a): perf score=0.007157
I20240430 09:42:59.392964 21672 raft_consensus.cc:482] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1aa766f8e66f4726b0d2adb9abb3c03f)
I20240430 09:42:59.393198 21672 raft_consensus.cc:504] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:59.394235 21672 leader_election.cc:288] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835), fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67:40877)
I20240430 09:42:59.395979 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "3e2dc1b121d04379820c7a44f613a47a" candidate_term: 2 candidate_status { last_received { term: 1 index: 996 } } ignore_live_leader: false dest_uuid: "fc63c0f78ed14575a782aab14a47a7a9" is_pre_election: true
W20240430 09:42:59.398067 21006 leader_election.cc:334] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111)
I20240430 09:42:59.398229 21006 leader_election.cc:302] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a; no voters: 1aa766f8e66f4726b0d2adb9abb3c03f, fc63c0f78ed14575a782aab14a47a7a9
I20240430 09:42:59.398957 21672 raft_consensus.cc:2726] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:42:59.400414 21480 raft_consensus.cc:482] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1aa766f8e66f4726b0d2adb9abb3c03f)
I20240430 09:42:59.400616 21480 raft_consensus.cc:504] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:59.401564 21480 leader_election.cc:288] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835)
I20240430 09:42:59.402992 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 996 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a" is_pre_election: true
I20240430 09:42:59.403270 21070 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc63c0f78ed14575a782aab14a47a7a9 in term 1.
I20240430 09:42:59.404188 21267 leader_election.cc:302] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9; no voters: 
I20240430 09:42:59.405879 21480 raft_consensus.cc:2781] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 09:42:59.406002 21480 raft_consensus.cc:482] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1aa766f8e66f4726b0d2adb9abb3c03f)
I20240430 09:42:59.406113 21480 raft_consensus.cc:3037] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 1 FOLLOWER]: Advancing to term 2
W20240430 09:42:59.406586 21267 leader_election.cc:334] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111)
I20240430 09:42:59.410301 21480 raft_consensus.cc:504] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:59.411732 21070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da44a9806d424d099f2397eeecb0a158" candidate_uuid: "fc63c0f78ed14575a782aab14a47a7a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 996 } } ignore_live_leader: false dest_uuid: "3e2dc1b121d04379820c7a44f613a47a"
I20240430 09:42:59.411986 21070 raft_consensus.cc:3037] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 1 FOLLOWER]: Advancing to term 2
I20240430 09:42:59.415748 21070 raft_consensus.cc:2454] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc63c0f78ed14575a782aab14a47a7a9 in term 2.
I20240430 09:42:59.416457 21267 leader_election.cc:302] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2dc1b121d04379820c7a44f613a47a, fc63c0f78ed14575a782aab14a47a7a9; no voters: 
I20240430 09:42:59.416882 21674 raft_consensus.cc:2781] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 09:42:59.417044 21674 raft_consensus.cc:686] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [term 2 LEADER]: Becoming Leader. State: Replica: fc63c0f78ed14575a782aab14a47a7a9, State: Running, Role: LEADER
I20240430 09:42:59.417431 21674 consensus_queue.cc:237] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 996, Committed index: 996, Last appended: 1.996, Last appended by leader: 996, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } }
I20240430 09:42:59.424713 20928 catalog_manager.cc:5484] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 reported cstate change: term changed from 1 to 2, leader changed from 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66) to fc63c0f78ed14575a782aab14a47a7a9 (127.20.101.67). New cstate: current_term: 2 leader_uuid: "fc63c0f78ed14575a782aab14a47a7a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1aa766f8e66f4726b0d2adb9abb3c03f" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc63c0f78ed14575a782aab14a47a7a9" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 40877 } health_report { overall_health: HEALTHY } } }
W20240430 09:42:59.429901 21267 leader_election.cc:334] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111)
I20240430 09:42:59.430260 21480 leader_election.cc:288] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [CANDIDATE]: Term 2 election: Requested vote from peers 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983), 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835)
W20240430 09:42:59.461673 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
W20240430 09:42:59.471274 21029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47264: Illegal state: replica 3e2dc1b121d04379820c7a44f613a47a is not leader of this config: current role FOLLOWER
I20240430 09:42:59.489344 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: MajorDeltaCompactionOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.137s	user 0.136s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5414,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":170,"lbm_read_time_us":357,"lbm_reads_lt_1ms":20,"lbm_write_time_us":348,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":529}
I20240430 09:42:59.490147 21377 maintenance_manager.cc:419] P fc63c0f78ed14575a782aab14a47a7a9: Scheduling UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158): 145637 bytes on disk
I20240430 09:42:59.490790 21268 maintenance_manager.cc:643] P fc63c0f78ed14575a782aab14a47a7a9: UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20240430 09:42:59.507114 21267 consensus_peers.cc:485] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 09:42:59.507953 21070 raft_consensus.cc:1261] T da44a9806d424d099f2397eeecb0a158 P 3e2dc1b121d04379820c7a44f613a47a [term 2 FOLLOWER]: Refusing update from remote peer fc63c0f78ed14575a782aab14a47a7a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 996. Preceding OpId from leader: term: 2 index: 998. (index mismatch)
I20240430 09:42:59.508862 21480 consensus_queue.cc:1035] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2dc1b121d04379820c7a44f613a47a" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 39983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 997, Last known committed idx: 996, Time since last communication: 0.000s
I20240430 09:43:00.071331 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: CompactRowSetsOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 1.611s	user 1.456s	sys 0.022s Metrics: {"bytes_written":578144,"cfile_cache_hit":12,"cfile_cache_hit_bytes":96550,"cfile_cache_miss":90,"cfile_cache_miss_bytes":2091992,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":13469,"dirs.run_cpu_time_us":1048,"dirs.run_wall_time_us":19350,"drs_written":1,"lbm_read_time_us":5315,"lbm_reads_lt_1ms":118,"lbm_write_time_us":6653,"lbm_writes_lt_1ms":76,"mutex_wait_us":37,"num_input_rowsets":2,"peak_mem_usage":1057721,"rows_written":32144,"thread_start_us":26120,"threads_started":7}
I20240430 09:43:00.075074 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling FlushMRSOp(da44a9806d424d099f2397eeecb0a158): perf score=1.000000
I20240430 09:43:00.262900 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022469858313760768 Inserted count: 51350
I20240430 09:43:00.263103 20885 linked_list-test-util.h:421] Finished inserting list. Added 51350 in chain
I20240430 09:43:00.263163 20885 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 09:43:00.263260 20885 linked_list-test-util.h:424] 0: -4056202606668152832
I20240430 09:43:00.263319 20885 linked_list-test-util.h:424] 1: -3555743714547269631
I20240430 09:43:00.263391 20885 linked_list-test-util.h:424] 2: -3055284822426386430
I20240430 09:43:00.263442 20885 linked_list-test-util.h:424] 3: -2554825930305503229
I20240430 09:43:00.263504 20885 linked_list-test-util.h:424] 4: -6110710382341062652
I20240430 09:43:00.263551 20885 linked_list-test-util.h:424] 5: -5610251490220179451
I20240430 09:43:00.263612 20885 linked_list-test-util.h:424] 6: -5109792598099296250
I20240430 09:43:00.263656 20885 linked_list-test-util.h:424] 7: -4609333705978413049
I20240430 09:43:00.263716 20885 linked_list-test-util.h:424] 8: -8165218158013972472
I20240430 09:43:00.263762 20885 linked_list-test-util.h:424] 9: -7664759265893089271
I20240430 09:43:00.263829 20885 linked_list-test-util.h:424] 10: -7164300373772206070
I20240430 09:43:00.263875 20885 linked_list-test-util.h:424] 11: -6663841481651322869
I20240430 09:43:00.263940 20885 linked_list-test-util.h:424] 12: 8227018140022669324
I20240430 09:43:00.263985 20885 linked_list-test-util.h:424] 13: 8727477032143552525
I20240430 09:43:00.264050 20885 linked_list-test-util.h:424] 14: -9218808149445115890
I20240430 09:43:00.264096 20885 linked_list-test-util.h:424] 15: -8718349257324232689
I20240430 09:43:00.264155 20885 linked_list-test-util.h:424] 16: 6172791839326470160
I20240430 09:43:00.264200 20885 linked_list-test-util.h:424] 17: 6673250731447353361
I20240430 09:43:00.264261 20885 linked_list-test-util.h:424] 18: 7173709623568236562
I20240430 09:43:00.264305 20885 linked_list-test-util.h:424] 19: 7674168515689119763
I20240430 09:43:00.264365 20885 linked_list-test-util.h:424] 20: 4118284063653560340
I20240430 09:43:00.264411 20885 linked_list-test-util.h:424] 21: 4618742955774443541
I20240430 09:43:00.264470 20885 linked_list-test-util.h:424] 22: 5119201847895326742
I20240430 09:43:00.264516 20885 linked_list-test-util.h:424] 23: 5619660740016209943
I20240430 09:43:00.264575 20885 linked_list-test-util.h:424] 24: 2063776287980650520
I20240430 09:43:00.264621 20885 linked_list-test-util.h:424] 25: 2564375917589823513
I20240430 09:43:00.264699 20885 linked_list-test-util.h:424] 26: 3064834809710706714
I20240430 09:43:00.264742 20885 linked_list-test-util.h:424] 27: -491049642324852709
I20240430 09:43:00.264807 20885 linked_list-test-util.h:424] 28: 9409249796030492
I20240430 09:43:00.264856 20885 linked_list-test-util.h:424] 29: 509868141916913693
I20240430 09:43:00.264914 20885 linked_list-test-util.h:424] 30: 1010327034037796894
I20240430 09:43:00.264958 20885 linked_list-test-util.h:424] 31: -2545275943021051873
I20240430 09:43:00.265015 20885 linked_list-test-util.h:424] 32: -2044817050900168672
I20240430 09:43:00.265077 20885 linked_list-test-util.h:424] 33: -1544358158779285471
I20240430 09:43:00.265136 20885 linked_list-test-util.h:424] 34: -1043899266658402270
I20240430 09:43:00.265182 20885 linked_list-test-util.h:424] 35: -4599783718693961693
I20240430 09:43:00.265239 20885 linked_list-test-util.h:424] 36: -4099324826573078492
I20240430 09:43:00.265285 20885 linked_list-test-util.h:424] 37: -3598865934452195291
I20240430 09:43:00.265349 20885 linked_list-test-util.h:424] 38: -3098407042331312090
I20240430 09:43:00.265394 20885 linked_list-test-util.h:424] 39: -6654291494366871513
I20240430 09:43:00.265453 20885 linked_list-test-util.h:424] 40: -6153832602245988312
I20240430 09:43:00.265499 20885 linked_list-test-util.h:424] 41: -5653373710125105111
I20240430 09:43:00.265559 20885 linked_list-test-util.h:424] 42: -5152914818004221910
I20240430 09:43:00.265605 20885 linked_list-test-util.h:424] 43: -8708799270039781333
I20240430 09:43:00.265668 20885 linked_list-test-util.h:424] 44: -8208340377918898132
I20240430 09:43:00.265714 20885 linked_list-test-util.h:424] 45: -7707881485798014931
I20240430 09:43:00.265774 20885 linked_list-test-util.h:424] 46: -7207422593677131730
I20240430 09:43:00.265826 20885 linked_list-test-util.h:424] 47: 7683718502973571119
I20240430 09:43:00.265887 20885 linked_list-test-util.h:424] 48: 8184177395094454320
I20240430 09:43:00.265931 20885 linked_list-test-util.h:424] 49: 8684636287215337521
I20240430 09:43:00.365965 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470160236713 usec, L: 1 Expected Rows: 19050
I20240430 09:43:00.387714 20885 linked_list-test-util.h:528] Calling callback on tserver fc63c0f78ed14575a782aab14a47a7a9
I20240430 09:43:00.565824 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: FlushMRSOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.490s	user 0.456s	sys 0.007s Metrics: {"bytes_written":1577,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1072,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2479,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8422,"update_count":218}
I20240430 09:43:00.566870 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling MajorDeltaCompactionOp(c980221eb2ae4752adb92064910d0161): perf score=0.017395
I20240430 09:43:00.637830 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: MajorDeltaCompactionOp(c980221eb2ae4752adb92064910d0161) complete. Timing: real 0.070s	user 0.050s	sys 0.007s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2819,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":5654,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1602,"lbm_read_time_us":332,"lbm_reads_lt_1ms":20,"lbm_write_time_us":372,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":13584,"threads_started":1,"update_count":300}
I20240430 09:43:00.639192 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling MajorDeltaCompactionOp(da44a9806d424d099f2397eeecb0a158): perf score=0.011885
I20240430 09:43:00.715224 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: MajorDeltaCompactionOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.076s	user 0.057s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2160,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1437,"lbm_read_time_us":818,"lbm_reads_lt_1ms":24,"lbm_write_time_us":3667,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"update_count":218}
I20240430 09:43:00.727231 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a): 609055 bytes on disk
I20240430 09:43:00.728015 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(2677af7c7e1e4846b54d331fa674a52a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240430 09:43:00.728855 21116 maintenance_manager.cc:419] P 3e2dc1b121d04379820c7a44f613a47a: Scheduling UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158): 158520 bytes on disk
I20240430 09:43:00.729714 21007 maintenance_manager.cc:643] P 3e2dc1b121d04379820c7a44f613a47a: UndoDeltaBlockGCOp(da44a9806d424d099f2397eeecb0a158) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240430 09:43:01.018401 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:43:01.028717 21006 consensus_peers.cc:485] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:43:01.043884 20885 linked_list-test-util.h:767] Done collecting results (19050 rows in 656.333ms)
I20240430 09:43:01.079629 20885 linked_list-test-util.h:556] Snapshot: P: 1714470160236713 usec, L: 1 verified. Result: OK
I20240430 09:43:01.082901 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470170268908 usec, L: 1 Expected Rows: 36800
I20240430 09:43:01.091023 20885 linked_list-test-util.h:528] Calling callback on tserver fc63c0f78ed14575a782aab14a47a7a9
W20240430 09:43:01.307077 21267 consensus_peers.cc:485] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:43:01.511008 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:43:01.733608 20885 linked_list-test-util.h:767] Done collecting results (36800 rows in 642.688ms)
I20240430 09:43:01.774917 20885 linked_list-test-util.h:556] Snapshot: P: 1714470170268908 usec, L: 1 verified. Result: OK
I20240430 09:43:01.777611 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470180252383 usec, L: 1 Expected Rows: 51350
I20240430 09:43:01.796352 20885 linked_list-test-util.h:528] Calling callback on tserver fc63c0f78ed14575a782aab14a47a7a9
W20240430 09:43:01.978538 21267 consensus_peers.cc:485] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:43:02.760802 21430 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 09:43:02.926733 20885 linked_list-test-util.h:767] Done collecting results (51350 rows in 1130.48ms)
I20240430 09:43:02.986856 20885 linked_list-test-util.h:556] Snapshot: P: 1714470180252383 usec, L: 1 verified. Result: OK
I20240430 09:43:02.989845 20885 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 51350
I20240430 09:43:03.260418 21452 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 09:43:03.519999 21267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 09:43:03.597643 21006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 09:43:03.598649 21006 consensus_peers.cc:485] T c980221eb2ae4752adb92064910d0161 P 3e2dc1b121d04379820c7a44f613a47a -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 09:43:03.910902 21267 consensus_peers.cc:485] T 2677af7c7e1e4846b54d331fa674a52a P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 09:43:04.204409 20885 linked_list-test-util.h:767] Done collecting results (51350 rows in 1205.14ms)
I20240430 09:43:04.265323 20885 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240430 09:43:04.265806 20885 linked_list-test-util.h:692] Successfully verified 51350 rows
W20240430 09:43:04.287721 21267 consensus_peers.cc:485] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 1aa766f8e66f4726b0d2adb9abb3c03f (127.20.101.66:43835): Couldn't send request to peer 1aa766f8e66f4726b0d2adb9abb3c03f. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 1aa766f8e66f4726b0d2adb9abb3c03f, Rpc address: host: "127.20.101.66" port: 43835 (Network error: Client connection negotiation failed: client connection to 127.20.101.66:43835: connect: Connection refused (error 111))
I20240430 09:43:04.303748 20885 webui_checker.cc:88] shutting down CURL thread
I20240430 09:43:04.504853 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 20988
I20240430 09:43:04.506178 21110 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:04.717963 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 20988
W20240430 09:43:04.775417 21267 connection.cc:535] client connection to 127.20.101.65:39983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 09:43:04.775758 20913 connection.cc:535] server connection from 127.20.101.65:57107 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 09:43:04.776018 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 21249
W20240430 09:43:04.776491 21267 consensus_peers.cc:485] T da44a9806d424d099f2397eeecb0a158 P fc63c0f78ed14575a782aab14a47a7a9 -> Peer 3e2dc1b121d04379820c7a44f613a47a (127.20.101.65:39983): Couldn't send request to peer 3e2dc1b121d04379820c7a44f613a47a. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:39983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 09:43:04.776988 21371 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:05.024856 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 21249
I20240430 09:43:05.082767 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 20897
I20240430 09:43:05.083937 20958 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:05.199610 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 20897
2024-04-30T09:43:05Z chronyd exiting
I20240430 09:43:05.240064 20885 test_util.cc:175] -----------------------------------------------
I20240430 09:43:05.240195 20885 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714470147653121-20885-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (37518 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240430 09:43:05.242282 20885 test_util.cc:268] Using random seed: 845057685
I20240430 09:43:05.244213 20885 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 09:43:05.244294 20885 linked_list-test.cc:101] --------------
I20240430 09:43:05.244365 20885 linked_list-test.cc:102] 50 chains
I20240430 09:43:05.244418 20885 linked_list-test.cc:103] 3 tablets
I20240430 09:43:05.244488 20885 linked_list-test.cc:104] Mutations on
I20240430 09:43:05.244545 20885 linked_list-test.cc:105] --------------
I20240430 09:43:05.244714 20885 ts_itest-base.cc:112] Starting cluster with:
I20240430 09:43:05.244771 20885 ts_itest-base.cc:113] --------------
I20240430 09:43:05.244843 20885 ts_itest-base.cc:114] 3 tablet servers
I20240430 09:43:05.244904 20885 ts_itest-base.cc:115] 3 replicas per TS
I20240430 09:43:05.244982 20885 ts_itest-base.cc:116] --------------
2024-04-30T09:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T09:43:05Z Disabled control of system clock
I20240430 09:43:05.277755 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.101.126:36401
--webserver_interface=127.20.101.126
--webserver_port=0
--builtin_ntp_servers=127.20.101.84:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.101.126:36401
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 09:43:05.620268 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:43:05.620579 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:43:05.620738 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:43:05.628558 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 09:43:05.628671 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:43:05.628727 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 09:43:05.628815 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 09:43:05.637840 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.101.126:36401
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.101.126:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.20.101.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:43:05.639259 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:43:05.641034 21741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:43:05.652827 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:05.653589 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:05.654248 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:43:05.654978 21741 server_base.cc:1034] running on GCE node
I20240430 09:43:05.655787 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:43:05.657193 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:43:05.658458 21741 hybrid_clock.cc:648] HybridClock initialized: now 1714470185658394 us; error 41 us; skew 500 ppm
I20240430 09:43:05.658769 21741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:43:05.661072 21741 webserver.cc:460] Webserver started at http://127.20.101.126:42139/ using document root <none> and password file <none>
I20240430 09:43:05.661764 21741 fs_manager.cc:359] Metadata directory not provided
I20240430 09:43:05.661896 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:43:05.662200 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:43:05.664871 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/instance:
uuid: "fb1648f57129407084b5f0ae220b17e5"
format_stamp: "Formatted at 2024-04-30 09:43:05 on dist-test-slave-8dh4"
I20240430 09:43:05.665606 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal/instance:
uuid: "fb1648f57129407084b5f0ae220b17e5"
format_stamp: "Formatted at 2024-04-30 09:43:05 on dist-test-slave-8dh4"
I20240430 09:43:05.671059 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240430 09:43:05.674479 21755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:05.675779 21741 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240430 09:43:05.675940 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
uuid: "fb1648f57129407084b5f0ae220b17e5"
format_stamp: "Formatted at 2024-04-30 09:43:05 on dist-test-slave-8dh4"
I20240430 09:43:05.676146 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:43:05.690670 21741 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:43:05.691498 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:43:05.691800 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:43:05.710224 21741 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.126:36401
I20240430 09:43:05.710242 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.126:36401 every 8 connection(s)
I20240430 09:43:05.711673 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/data/info.pb
I20240430 09:43:05.715485 21807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:43:05.719228 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 21741
I20240430 09:43:05.719439 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/master-0/wal/instance
I20240430 09:43:05.725984 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5: Bootstrap starting.
I20240430 09:43:05.728950 21807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5: Neither blocks nor log segments found. Creating new log.
I20240430 09:43:05.730150 21807 log.cc:824] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5: Log is configured to *not* fsync() on all Append() calls
I20240430 09:43:05.733068 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5: No bootstrap required, opened a new log
I20240430 09:43:05.739594 21807 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } }
I20240430 09:43:05.739934 21807 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:43:05.740036 21807 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1648f57129407084b5f0ae220b17e5, State: Initialized, Role: FOLLOWER
I20240430 09:43:05.740597 21807 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } }
I20240430 09:43:05.740808 21807 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 09:43:05.740947 21807 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 09:43:05.741096 21807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:43:05.743302 21807 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } }
I20240430 09:43:05.743769 21807 leader_election.cc:302] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb1648f57129407084b5f0ae220b17e5; no voters: 
I20240430 09:43:05.744304 21807 leader_election.cc:288] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 09:43:05.744454 21812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:43:05.745045 21812 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [term 1 LEADER]: Becoming Leader. State: Replica: fb1648f57129407084b5f0ae220b17e5, State: Running, Role: LEADER
I20240430 09:43:05.745592 21812 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } }
I20240430 09:43:05.746215 21807 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 09:43:05.748466 21813 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb1648f57129407084b5f0ae220b17e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } } }
I20240430 09:43:05.748531 21814 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb1648f57129407084b5f0ae220b17e5. Latest consensus state: current_term: 1 leader_uuid: "fb1648f57129407084b5f0ae220b17e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1648f57129407084b5f0ae220b17e5" member_type: VOTER last_known_addr { host: "127.20.101.126" port: 36401 } } }
I20240430 09:43:05.748899 21814 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [sys.catalog]: This master's current role is: LEADER
I20240430 09:43:05.748898 21813 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5 [sys.catalog]: This master's current role is: LEADER
I20240430 09:43:05.749760 21817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 09:43:05.759778 21817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 09:43:05.769634 21817 catalog_manager.cc:1347] Generated new cluster ID: 592d8467269143648190cb6239aa48f0
I20240430 09:43:05.769768 21817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 09:43:05.791599 21817 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 09:43:05.793349 21817 catalog_manager.cc:1504] Loading token signing keys...
I20240430 09:43:05.803115 21817 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fb1648f57129407084b5f0ae220b17e5: Generated new TSK 0
I20240430 09:43:05.804211 21817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 09:43:05.811086 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.65:0
--local_ip_for_outbound_sockets=127.20.101.65
--webserver_interface=127.20.101.65
--webserver_port=0
--tserver_master_addrs=127.20.101.126:36401
--builtin_ntp_servers=127.20.101.84:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 09:43:06.141793 21831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:43:06.142130 21831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:43:06.142299 21831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:43:06.149978 21831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:43:06.150234 21831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.65
I20240430 09:43:06.159545 21831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.101.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.101.126:36401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.65
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:43:06.160923 21831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:43:06.162707 21831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:43:06.174767 21836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:06.175410 21839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:06.176048 21837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:43:06.177000 21831 server_base.cc:1034] running on GCE node
I20240430 09:43:06.177554 21831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:43:06.178504 21831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:43:06.179720 21831 hybrid_clock.cc:648] HybridClock initialized: now 1714470186179657 us; error 40 us; skew 500 ppm
I20240430 09:43:06.180025 21831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:43:06.182406 21831 webserver.cc:460] Webserver started at http://127.20.101.65:39085/ using document root <none> and password file <none>
I20240430 09:43:06.183146 21831 fs_manager.cc:359] Metadata directory not provided
I20240430 09:43:06.183275 21831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:43:06.183583 21831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:43:06.186224 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/instance:
uuid: "8d65d5b97d40479c8d0d65c30dda3557"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.187007 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/instance:
uuid: "8d65d5b97d40479c8d0d65c30dda3557"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.192111 21831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240430 09:43:06.195626 21845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.196970 21831 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240430 09:43:06.197127 21831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
uuid: "8d65d5b97d40479c8d0d65c30dda3557"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.197290 21831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:43:06.236249 21831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:43:06.237133 21831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:43:06.237434 21831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:43:06.238355 21831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:43:06.239874 21831 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:43:06.239945 21831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.240067 21831 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:43:06.240121 21831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.270416 21831 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.65:41927
I20240430 09:43:06.270486 21957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.65:41927 every 8 connection(s)
I20240430 09:43:06.271749 21831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
I20240430 09:43:06.273015 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 21831
I20240430 09:43:06.273253 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/instance
I20240430 09:43:06.277101 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.66:0
--local_ip_for_outbound_sockets=127.20.101.66
--webserver_interface=127.20.101.66
--webserver_port=0
--tserver_master_addrs=127.20.101.126:36401
--builtin_ntp_servers=127.20.101.84:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240430 09:43:06.286947 21958 heartbeater.cc:347] Connected to a master server at 127.20.101.126:36401
I20240430 09:43:06.287237 21958 heartbeater.cc:464] Registering TS with master...
I20240430 09:43:06.288080 21958 heartbeater.cc:511] Master 127.20.101.126:36401 requested a full tablet report, sending...
I20240430 09:43:06.290349 21772 ts_manager.cc:194] Registered new tserver with Master: 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927)
I20240430 09:43:06.292547 21772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.65:56027
W20240430 09:43:06.612200 21962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:43:06.612504 21962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:43:06.612609 21962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:43:06.620244 21962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:43:06.620465 21962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.66
I20240430 09:43:06.629465 21962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.20.101.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.101.126:36401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.66
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:43:06.630895 21962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:43:06.632617 21962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:43:06.645462 21967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:06.646063 21968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:06.646962 21970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:43:06.647775 21962 server_base.cc:1034] running on GCE node
I20240430 09:43:06.648471 21962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:43:06.649452 21962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:43:06.650684 21962 hybrid_clock.cc:648] HybridClock initialized: now 1714470186650602 us; error 53 us; skew 500 ppm
I20240430 09:43:06.651031 21962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:43:06.653370 21962 webserver.cc:460] Webserver started at http://127.20.101.66:33189/ using document root <none> and password file <none>
I20240430 09:43:06.654099 21962 fs_manager.cc:359] Metadata directory not provided
I20240430 09:43:06.654232 21962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:43:06.654573 21962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:43:06.657187 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/instance:
uuid: "2f3719f20ee14a02a13b62ef7d239d53"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.657905 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal/instance:
uuid: "2f3719f20ee14a02a13b62ef7d239d53"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.663069 21962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240430 09:43:06.666249 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.667589 21962 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240430 09:43:06.667752 21962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
uuid: "2f3719f20ee14a02a13b62ef7d239d53"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:06.667919 21962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:43:06.688225 21962 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:43:06.689121 21962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:43:06.689548 21962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:43:06.690652 21962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:43:06.692487 21962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:43:06.692572 21962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.692715 21962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:43:06.692790 21962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:06.724793 21962 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.66:39963
I20240430 09:43:06.724794 22088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.66:39963 every 8 connection(s)
I20240430 09:43:06.726461 21962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/data/info.pb
I20240430 09:43:06.733319 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 21962
I20240430 09:43:06.733489 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-1/wal/instance
I20240430 09:43:06.736538 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.67:0
--local_ip_for_outbound_sockets=127.20.101.67
--webserver_interface=127.20.101.67
--webserver_port=0
--tserver_master_addrs=127.20.101.126:36401
--builtin_ntp_servers=127.20.101.84:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240430 09:43:06.739696 22089 heartbeater.cc:347] Connected to a master server at 127.20.101.126:36401
I20240430 09:43:06.739980 22089 heartbeater.cc:464] Registering TS with master...
I20240430 09:43:06.740797 22089 heartbeater.cc:511] Master 127.20.101.126:36401 requested a full tablet report, sending...
I20240430 09:43:06.742549 21772 ts_manager.cc:194] Registered new tserver with Master: 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963)
I20240430 09:43:06.743376 21772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.66:48969
W20240430 09:43:07.070475 22092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:43:07.070829 22092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:43:07.070995 22092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:43:07.078629 22092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:43:07.078857 22092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.67
I20240430 09:43:07.088050 22092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.20.101.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.101.126:36401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.67
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:43:07.089676 22092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:43:07.091670 22092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:43:07.104498 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:07.105192 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:07.105597 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:43:07.106562 22092 server_base.cc:1034] running on GCE node
I20240430 09:43:07.107192 22092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:43:07.108107 22092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:43:07.109329 22092 hybrid_clock.cc:648] HybridClock initialized: now 1714470187109257 us; error 43 us; skew 500 ppm
I20240430 09:43:07.109637 22092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:43:07.112022 22092 webserver.cc:460] Webserver started at http://127.20.101.67:43777/ using document root <none> and password file <none>
I20240430 09:43:07.112754 22092 fs_manager.cc:359] Metadata directory not provided
I20240430 09:43:07.112890 22092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:43:07.113188 22092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 09:43:07.116101 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/instance:
uuid: "68003d097d54450cb9ddf0d0c753eb0e"
format_stamp: "Formatted at 2024-04-30 09:43:07 on dist-test-slave-8dh4"
I20240430 09:43:07.116827 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal/instance:
uuid: "68003d097d54450cb9ddf0d0c753eb0e"
format_stamp: "Formatted at 2024-04-30 09:43:07 on dist-test-slave-8dh4"
I20240430 09:43:07.122045 22092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240430 09:43:07.125516 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:07.126978 22092 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240430 09:43:07.127151 22092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
uuid: "68003d097d54450cb9ddf0d0c753eb0e"
format_stamp: "Formatted at 2024-04-30 09:43:07 on dist-test-slave-8dh4"
I20240430 09:43:07.127327 22092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:43:07.162619 22092 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:43:07.163460 22092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:43:07.163750 22092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:43:07.164732 22092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:43:07.166224 22092 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 09:43:07.166311 22092 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:07.166451 22092 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 09:43:07.166507 22092 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:07.198123 22092 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.67:36427
I20240430 09:43:07.198181 22218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.67:36427 every 8 connection(s)
I20240430 09:43:07.199455 22092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/data/info.pb
I20240430 09:43:07.208905 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 22092
I20240430 09:43:07.209162 20885 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-2/wal/instance
I20240430 09:43:07.212033 22219 heartbeater.cc:347] Connected to a master server at 127.20.101.126:36401
I20240430 09:43:07.212313 22219 heartbeater.cc:464] Registering TS with master...
I20240430 09:43:07.213074 22219 heartbeater.cc:511] Master 127.20.101.126:36401 requested a full tablet report, sending...
I20240430 09:43:07.214403 21772 ts_manager.cc:194] Registered new tserver with Master: 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427)
I20240430 09:43:07.215198 21772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.67:58357
I20240430 09:43:07.224700 20885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:43:07.239399 21772 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54898:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 09:43:07.245369 21772 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 09:43:07.269863 22154 tablet_service.cc:1467] Processing CreateTablet for tablet f9388a35737a46d2bd003dddb7efe1f0 (DEFAULT_TABLE table=TestTable [id=c40b4bdc7586449fb6bfd27fd2c07c42]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 09:43:07.270030 21893 tablet_service.cc:1467] Processing CreateTablet for tablet f9388a35737a46d2bd003dddb7efe1f0 (DEFAULT_TABLE table=TestTable [id=c40b4bdc7586449fb6bfd27fd2c07c42]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 09:43:07.270015 22024 tablet_service.cc:1467] Processing CreateTablet for tablet f9388a35737a46d2bd003dddb7efe1f0 (DEFAULT_TABLE table=TestTable [id=c40b4bdc7586449fb6bfd27fd2c07c42]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 09:43:07.271447 21893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9388a35737a46d2bd003dddb7efe1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:43:07.271466 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9388a35737a46d2bd003dddb7efe1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:43:07.271652 22154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9388a35737a46d2bd003dddb7efe1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 09:43:07.284392 22237 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Bootstrap starting.
I20240430 09:43:07.286286 22238 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Bootstrap starting.
I20240430 09:43:07.287079 22239 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Bootstrap starting.
I20240430 09:43:07.288651 22237 tablet_bootstrap.cc:654] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Neither blocks nor log segments found. Creating new log.
I20240430 09:43:07.289983 22238 tablet_bootstrap.cc:654] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Neither blocks nor log segments found. Creating new log.
I20240430 09:43:07.289969 22237 log.cc:824] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Log is configured to *not* fsync() on all Append() calls
I20240430 09:43:07.290380 22239 tablet_bootstrap.cc:654] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Neither blocks nor log segments found. Creating new log.
I20240430 09:43:07.291162 22238 log.cc:824] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Log is configured to *not* fsync() on all Append() calls
I20240430 09:43:07.291473 22239 log.cc:824] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Log is configured to *not* fsync() on all Append() calls
I20240430 09:43:07.293208 22238 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: No bootstrap required, opened a new log
I20240430 09:43:07.293192 22237 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: No bootstrap required, opened a new log
I20240430 09:43:07.293474 22237 ts_tablet_manager.cc:1367] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240430 09:43:07.293481 22238 ts_tablet_manager.cc:1367] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240430 09:43:07.295311 22239 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: No bootstrap required, opened a new log
I20240430 09:43:07.295559 22239 ts_tablet_manager.cc:1367] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240430 09:43:07.295588 21958 heartbeater.cc:503] Master 127.20.101.126:36401 was elected leader, sending a full tablet report...
I20240430 09:43:07.302006 22237 raft_consensus.cc:348] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.302034 22238 raft_consensus.cc:348] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.302541 22237 raft_consensus.cc:374] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:43:07.302665 22238 raft_consensus.cc:374] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:43:07.302692 22237 raft_consensus.cc:729] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68003d097d54450cb9ddf0d0c753eb0e, State: Initialized, Role: FOLLOWER
I20240430 09:43:07.302819 22238 raft_consensus.cc:729] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3719f20ee14a02a13b62ef7d239d53, State: Initialized, Role: FOLLOWER
I20240430 09:43:07.303404 22238 consensus_queue.cc:260] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.303408 22237 consensus_queue.cc:260] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.304410 22089 heartbeater.cc:503] Master 127.20.101.126:36401 was elected leader, sending a full tablet report...
I20240430 09:43:07.304361 22239 raft_consensus.cc:348] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.304854 22239 raft_consensus.cc:374] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 09:43:07.304982 22239 raft_consensus.cc:729] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d65d5b97d40479c8d0d65c30dda3557, State: Initialized, Role: FOLLOWER
I20240430 09:43:07.305702 22239 consensus_queue.cc:260] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.306305 22238 ts_tablet_manager.cc:1398] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.008s
I20240430 09:43:07.309554 22239 ts_tablet_manager.cc:1398] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.002s
I20240430 09:43:07.309950 22237 ts_tablet_manager.cc:1398] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Time spent starting tablet: real 0.016s	user 0.013s	sys 0.005s
I20240430 09:43:07.310108 22219 heartbeater.cc:503] Master 127.20.101.126:36401 was elected leader, sending a full tablet report...
W20240430 09:43:07.451063 22220 tablet.cc:2369] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 09:43:07.478554 22090 tablet.cc:2369] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 09:43:07.524250 21959 tablet.cc:2369] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:43:07.568367 22243 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 09:43:07.568639 22243 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.570281 22243 leader_election.cc:288] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427), 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927)
I20240430 09:43:07.577601 22174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9388a35737a46d2bd003dddb7efe1f0" candidate_uuid: "2f3719f20ee14a02a13b62ef7d239d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68003d097d54450cb9ddf0d0c753eb0e" is_pre_election: true
I20240430 09:43:07.577919 21913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9388a35737a46d2bd003dddb7efe1f0" candidate_uuid: "2f3719f20ee14a02a13b62ef7d239d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d65d5b97d40479c8d0d65c30dda3557" is_pre_election: true
I20240430 09:43:07.578088 22174 raft_consensus.cc:2454] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3719f20ee14a02a13b62ef7d239d53 in term 0.
I20240430 09:43:07.578361 21913 raft_consensus.cc:2454] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3719f20ee14a02a13b62ef7d239d53 in term 0.
I20240430 09:43:07.578712 21977 leader_election.cc:302] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3719f20ee14a02a13b62ef7d239d53, 68003d097d54450cb9ddf0d0c753eb0e; no voters: 
I20240430 09:43:07.579098 22243 raft_consensus.cc:2781] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 09:43:07.579242 22243 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 09:43:07.579352 22243 raft_consensus.cc:3037] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:43:07.581481 22243 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.582039 22243 leader_election.cc:288] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [CANDIDATE]: Term 1 election: Requested vote from peers 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427), 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927)
I20240430 09:43:07.582489 21913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9388a35737a46d2bd003dddb7efe1f0" candidate_uuid: "2f3719f20ee14a02a13b62ef7d239d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d65d5b97d40479c8d0d65c30dda3557"
I20240430 09:43:07.582504 22174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9388a35737a46d2bd003dddb7efe1f0" candidate_uuid: "2f3719f20ee14a02a13b62ef7d239d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68003d097d54450cb9ddf0d0c753eb0e"
I20240430 09:43:07.582648 21913 raft_consensus.cc:3037] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:43:07.582674 22174 raft_consensus.cc:3037] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 0 FOLLOWER]: Advancing to term 1
I20240430 09:43:07.584672 22174 raft_consensus.cc:2454] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3719f20ee14a02a13b62ef7d239d53 in term 1.
I20240430 09:43:07.584692 21913 raft_consensus.cc:2454] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3719f20ee14a02a13b62ef7d239d53 in term 1.
I20240430 09:43:07.585063 21977 leader_election.cc:302] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3719f20ee14a02a13b62ef7d239d53, 68003d097d54450cb9ddf0d0c753eb0e; no voters: 
I20240430 09:43:07.585381 22243 raft_consensus.cc:2781] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 09:43:07.585762 22243 raft_consensus.cc:686] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3719f20ee14a02a13b62ef7d239d53, State: Running, Role: LEADER
I20240430 09:43:07.586316 22243 consensus_queue.cc:237] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:07.590735 21771 catalog_manager.cc:5484] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66). New cstate: current_term: 1 leader_uuid: "2f3719f20ee14a02a13b62ef7d239d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } health_report { overall_health: HEALTHY } } }
I20240430 09:43:07.626242 20885 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 09:43:07.628700 20885 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8d65d5b97d40479c8d0d65c30dda3557 to finish bootstrapping
I20240430 09:43:07.637540 20885 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2f3719f20ee14a02a13b62ef7d239d53 to finish bootstrapping
I20240430 09:43:07.645264 20885 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 68003d097d54450cb9ddf0d0c753eb0e to finish bootstrapping
I20240430 09:43:07.652710 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 21831
I20240430 09:43:07.653681 21953 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:07.747203 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 21831
I20240430 09:43:07.768301 20885 linked_list-test-util.h:401] Running for: 30.000s
I20240430 09:43:07.768445 20885 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 09:43:07.768507 20885 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
W20240430 09:43:07.790201 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111)
I20240430 09:43:07.790846 22174 raft_consensus.cc:1261] T f9388a35737a46d2bd003dddb7efe1f0 P 68003d097d54450cb9ddf0d0c753eb0e [term 1 FOLLOWER]: Refusing update from remote peer 2f3719f20ee14a02a13b62ef7d239d53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240430 09:43:07.791808 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 09:43:07.791827 22243 consensus_queue.cc:1035] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 09:43:07.799784 22255 mvcc.cc:205] Tried to move back new op lower bound from 7022469889180377088 to 7022469888357396480. Current Snapshot: MvccSnapshot[applied={T|T < 7022469889180377088}]
W20240430 09:43:10.165585 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 09:43:12.726671 22261 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 47 ops behind the committed index 
W20240430 09:43:12.740360 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 09:43:13.135084 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 09:43:15.159159 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 09:43:16.171316 22243 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 391 ops behind the committed index  [suppressed 6 similar messages]
W20240430 09:43:17.707717 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 09:43:17.768954 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022469930042253312 Inserted count: 52150
W20240430 09:43:18.229207 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 09:43:19.641885 22309 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 739 ops behind the committed index  [suppressed 6 similar messages]
W20240430 09:43:20.027730 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 09:43:22.747799 22337 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 1047 ops behind the committed index  [suppressed 5 similar messages]
W20240430 09:43:22.765878 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 09:43:23.563042 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 09:43:25.144093 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 09:43:26.111596 22309 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 1393 ops behind the committed index  [suppressed 6 similar messages]
W20240430 09:43:27.743263 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240430 09:43:27.770279 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022469971008237568 Inserted count: 101650
W20240430 09:43:28.801848 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 09:43:29.242864 22366 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 1703 ops behind the committed index  [suppressed 5 similar messages]
W20240430 09:43:30.228626 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 09:43:32.793344 22366 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 2059 ops behind the committed index  [suppressed 6 similar messages]
W20240430 09:43:32.801774 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 09:43:34.025862 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240430 09:43:35.336292 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 09:43:36.326258 22427 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 2415 ops behind the committed index  [suppressed 6 similar messages]
I20240430 09:43:37.774730 20885 linked_list-test-util.h:417] Sample at HT timestamp: 7022470011986432000 Inserted count: 151300
I20240430 09:43:37.774875 20885 linked_list-test-util.h:421] Finished inserting list. Added 151300 in chain
I20240430 09:43:37.774936 20885 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 09:43:37.775012 20885 linked_list-test-util.h:424] 0: 8926461433352093696
I20240430 09:43:37.775084 20885 linked_list-test-util.h:424] 1: -6986361068192661503
I20240430 09:43:37.775157 20885 linked_list-test-util.h:424] 2: -4452580233516154878
I20240430 09:43:37.775207 20885 linked_list-test-util.h:424] 3: -1918799398839648253
I20240430 09:43:37.775290 20885 linked_list-test-util.h:424] 4: -8905019729543888892
I20240430 09:43:37.775347 20885 linked_list-test-util.h:424] 5: -6371098157379092475
I20240430 09:43:37.775415 20885 linked_list-test-util.h:424] 6: -3837317322702585850
I20240430 09:43:37.775466 20885 linked_list-test-util.h:424] 7: -1303395750537789433
I20240430 09:43:37.775532 20885 linked_list-test-util.h:424] 8: -8289616081242030072
I20240430 09:43:37.775583 20885 linked_list-test-util.h:424] 9: -5755835246565523447
I20240430 09:43:37.775650 20885 linked_list-test-util.h:424] 10: -3221913674400727030
I20240430 09:43:37.775700 20885 linked_list-test-util.h:424] 11: -688132839724220405
I20240430 09:43:37.775764 20885 linked_list-test-util.h:424] 12: -7674634645405171700
I20240430 09:43:37.775817 20885 linked_list-test-util.h:424] 13: -5140853810728665075
I20240430 09:43:37.775882 20885 linked_list-test-util.h:424] 14: -2606932238563868658
I20240430 09:43:37.775933 20885 linked_list-test-util.h:424] 15: -73151403887362033
I20240430 09:43:37.775997 20885 linked_list-test-util.h:424] 16: -7059371734591602672
I20240430 09:43:37.776053 20885 linked_list-test-util.h:424] 17: -4525450162426806255
I20240430 09:43:37.776119 20885 linked_list-test-util.h:424] 18: -1991669327750299630
I20240430 09:43:37.776170 20885 linked_list-test-util.h:424] 19: 542252244414496787
I20240430 09:43:37.776245 20885 linked_list-test-util.h:424] 20: -6443968086289743852
I20240430 09:43:37.776302 20885 linked_list-test-util.h:424] 21: -3910187251613237227
I20240430 09:43:37.776364 20885 linked_list-test-util.h:424] 22: -1376406416936730602
I20240430 09:43:37.776413 20885 linked_list-test-util.h:424] 23: 1157515155228065815
I20240430 09:43:37.776476 20885 linked_list-test-util.h:424] 24: -5828986650452885480
I20240430 09:43:37.776525 20885 linked_list-test-util.h:424] 25: -3295205815776378855
I20240430 09:43:37.776589 20885 linked_list-test-util.h:424] 26: -761002768634871782
I20240430 09:43:37.776644 20885 linked_list-test-util.h:424] 27: -7747504574315823077
I20240430 09:43:37.776713 20885 linked_list-test-util.h:424] 28: -5213723739639316452
I20240430 09:43:37.776763 20885 linked_list-test-util.h:424] 29: -2679802167474520035
I20240430 09:43:37.776845 20885 linked_list-test-util.h:424] 30: -146021332798013410
I20240430 09:43:37.776899 20885 linked_list-test-util.h:424] 31: -7132241663502254049
I20240430 09:43:37.776968 20885 linked_list-test-util.h:424] 32: -4598320091337457632
I20240430 09:43:37.777019 20885 linked_list-test-util.h:424] 33: -2064539256660951007
I20240430 09:43:37.777098 20885 linked_list-test-util.h:424] 34: 469241578015555618
I20240430 09:43:37.777153 20885 linked_list-test-util.h:424] 35: -6516978752688685021
I20240430 09:43:37.777220 20885 linked_list-test-util.h:424] 36: -3983057180523888604
I20240430 09:43:37.777271 20885 linked_list-test-util.h:424] 37: -1449276345847381979
I20240430 09:43:37.777333 20885 linked_list-test-util.h:424] 38: 1084645226317414438
I20240430 09:43:37.777379 20885 linked_list-test-util.h:424] 39: -5901856579363536857
I20240430 09:43:37.777438 20885 linked_list-test-util.h:424] 40: -3368075744687030232
I20240430 09:43:37.777483 20885 linked_list-test-util.h:424] 41: -834154172522233815
I20240430 09:43:37.777544 20885 linked_list-test-util.h:424] 42: 1699626662154272810
I20240430 09:43:37.777591 20885 linked_list-test-util.h:424] 43: -5286593668549967829
I20240430 09:43:37.777652 20885 linked_list-test-util.h:424] 44: -2752812833873461204
I20240430 09:43:37.777699 20885 linked_list-test-util.h:424] 45: -218891261708664787
I20240430 09:43:37.777758 20885 linked_list-test-util.h:424] 46: 2314889572967841838
I20240430 09:43:37.777804 20885 linked_list-test-util.h:424] 47: -4671330757736398801
I20240430 09:43:37.777865 20885 linked_list-test-util.h:424] 48: -2137409185571602384
I20240430 09:43:37.777911 20885 linked_list-test-util.h:424] 49: 396371649104904241
I20240430 09:43:37.827169 20885 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGww_u/build/asan/bin/kudu
/tmp/dist-test-taskGGww_u/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.101.65:41927
--local_ip_for_outbound_sockets=127.20.101.65
--tserver_master_addrs=127.20.101.126:36401
--webserver_port=39085
--webserver_interface=127.20.101.65
--builtin_ntp_servers=127.20.101.84:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 09:43:37.854977 21978 consensus_peers.cc:485] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 -> Peer 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927): Couldn't send request to peer 8d65d5b97d40479c8d0d65c30dda3557. Status: Network error: Client connection negotiation failed: client connection to 127.20.101.65:41927: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 09:43:38.191998 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 09:43:38.192328 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 09:43:38.192477 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 09:43:38.200249 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 09:43:38.200475 22453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.101.65
I20240430 09:43:38.210005 22453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.101.84:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.101.65:41927
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.101.65
--webserver_port=39085
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.101.126:36401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.101.65
--log_dir=/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 09:13:05 UTC on efd1337a2aa3
build id 51
ASAN enabled
I20240430 09:43:38.211455 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 09:43:38.213279 22453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 09:43:38.225689 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:38.226322 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 09:43:38.227042 22463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 09:43:38.227957 22453 server_base.cc:1034] running on GCE node
I20240430 09:43:38.228565 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 09:43:38.229620 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 09:43:38.230829 22453 hybrid_clock.cc:648] HybridClock initialized: now 1714470218230756 us; error 52 us; skew 500 ppm
I20240430 09:43:38.231153 22453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 09:43:38.233479 22453 webserver.cc:460] Webserver started at http://127.20.101.65:39085/ using document root <none> and password file <none>
I20240430 09:43:38.234179 22453 fs_manager.cc:359] Metadata directory not provided
I20240430 09:43:38.234318 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 09:43:38.239611 22453 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240430 09:43:38.242448 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 09:43:38.243871 22453 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240430 09:43:38.244026 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
uuid: "8d65d5b97d40479c8d0d65c30dda3557"
format_stamp: "Formatted at 2024-04-30 09:43:06 on dist-test-slave-8dh4"
I20240430 09:43:38.244882 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 09:43:38.279131 22453 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 09:43:38.280026 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 09:43:38.280299 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 09:43:38.281287 22453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 09:43:38.283181 22476 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240430 09:43:38.288462 22453 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240430 09:43:38.288585 22453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
I20240430 09:43:38.288717 22453 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240430 09:43:38.291603 22453 ts_tablet_manager.cc:590] Registered 1 tablets
I20240430 09:43:38.291682 22453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.000s	sys 0.002s
I20240430 09:43:38.292035 22476 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Bootstrap starting.
I20240430 09:43:38.315537 22476 log.cc:824] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Log is configured to *not* fsync() on all Append() calls
I20240430 09:43:38.319530 22476 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 09:43:38.319883 22476 log.cc:1225] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/wal/wals/f9388a35737a46d2bd003dddb7efe1f0.recovery-1714470218319749
I20240430 09:43:38.319983 22476 tablet_bootstrap.cc:492] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Bootstrap complete.
I20240430 09:43:38.320537 22476 ts_tablet_manager.cc:1367] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Time spent bootstrapping tablet: real 0.029s	user 0.021s	sys 0.004s
I20240430 09:43:38.327824 22583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.101.65:41927 every 8 connection(s)
I20240430 09:43:38.327821 22453 rpc_server.cc:307] RPC server started. Bound to: 127.20.101.65:41927
I20240430 09:43:38.329866 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGww_u/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714470147653121-20885-0/linked-list-cluster/ts-0/data/info.pb
I20240430 09:43:38.333469 22476 raft_consensus.cc:348] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:38.334782 22476 raft_consensus.cc:729] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d65d5b97d40479c8d0d65c30dda3557, State: Initialized, Role: FOLLOWER
I20240430 09:43:38.335389 22476 consensus_queue.cc:260] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:38.336798 22476 ts_tablet_manager.cc:1398] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Time spent starting tablet: real 0.016s	user 0.017s	sys 0.000s
I20240430 09:43:38.338917 20885 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGww_u/build/asan/bin/kudu as pid 22453
I20240430 09:43:38.343096 22584 heartbeater.cc:347] Connected to a master server at 127.20.101.126:36401
I20240430 09:43:38.343398 22584 heartbeater.cc:464] Registering TS with master...
I20240430 09:43:38.344230 22584 heartbeater.cc:511] Master 127.20.101.126:36401 requested a full tablet report, sending...
I20240430 09:43:38.347074 21765 ts_manager.cc:194] Re-registered known tserver with Master: 8d65d5b97d40479c8d0d65c30dda3557 (127.20.101.65:41927)
I20240430 09:43:38.349257 21765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.101.65:37453
I20240430 09:43:38.351989 22584 heartbeater.cc:503] Master 127.20.101.126:36401 was elected leader, sending a full tablet report...
I20240430 09:43:38.372869 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 0.0
I20240430 09:43:38.476202 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 0.0
W20240430 09:43:38.581800 22585 tablet.cc:2369] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 09:43:38.679471 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 0.0
I20240430 09:43:38.990846 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 1.495
I20240430 09:43:39.453012 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 1.1488
I20240430 09:43:39.543581 22427 consensus_queue.cc:786] T f9388a35737a46d2bd003dddb7efe1f0 P 2f3719f20ee14a02a13b62ef7d239d53 [LEADER]: Peer 8d65d5b97d40479c8d0d65c30dda3557 is lagging by at least 1064 ops behind the committed index  [suppressed 7 similar messages]
I20240430 09:43:39.778936 22595 mvcc.cc:205] Tried to move back new op lower bound from 7022469907828645888 to 7022469888357396480. Current Snapshot: MvccSnapshot[applied={T|T < 7022469906251157504 or (T in {7022469906258239488,7022469906309332992,7022469906354876416,7022469906398883840,7022469906443485184,7022469906487934976,7022469906534551552,7022469906591535104,7022469906649006080,7022469906733137920,7022469906771603456,7022469906818834432,7022469906860814336,7022469906896347136,7022469906933911552,7022469906966802432,7022469907002871808,7022469907033669632,7022469907069308928,7022469907131117568,7022469907100418048,7022469907200000000,7022469907276349440,7022469907234357248,7022469907163230208,7022469907373670400,7022469907341189120,7022469907405074432,7022469907436736512,7022469907468247040,7022469907309043712,7022469907500122112,7022469907596582912})}]
I20240430 09:43:39.984939 20885 cluster_itest_util.cc:256] Not converged past 3026 yet: 1.3057 1.3057 1.2466
I20240430 09:43:40.588773 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 21962
I20240430 09:43:40.589845 22084 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:40.914867 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 21962
I20240430 09:43:40.990965 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 22092
I20240430 09:43:40.992172 22214 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:41.292184 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 22092
I20240430 09:43:41.355284 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470197764222 usec, L: 1 Expected Rows: 52150
I20240430 09:43:41.369206 20885 meta_cache.cc:1510] marking tablet server 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) as failed
W20240430 09:43:41.369465 20885 meta_cache.cc:302] tablet f9388a35737a46d2bd003dddb7efe1f0: replica 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:41.449399 20885 linked_list-test-util.h:528] Calling callback on tserver 8d65d5b97d40479c8d0d65c30dda3557
I20240430 09:43:42.401110 22609 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3719f20ee14a02a13b62ef7d239d53)
I20240430 09:43:42.401392 22609 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:42.403055 22609 leader_election.cc:288] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427), 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963)
W20240430 09:43:42.404251 22470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.67:36427: connect: Connection refused (error 111)
W20240430 09:43:42.406224 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427): Network error: Client connection negotiation failed: client connection to 127.20.101.67:36427: connect: Connection refused (error 111)
W20240430 09:43:42.406477 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963): Network error: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:42.406571 22470 leader_election.cc:302] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d65d5b97d40479c8d0d65c30dda3557; no voters: 2f3719f20ee14a02a13b62ef7d239d53, 68003d097d54450cb9ddf0d0c753eb0e
I20240430 09:43:42.406926 22609 raft_consensus.cc:2726] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:43:43.502787 20885 linked_list-test-util.h:767] Done collecting results (52150 rows in 2053.53ms)
I20240430 09:43:43.612597 20885 linked_list-test-util.h:556] Snapshot: P: 1714470197764222 usec, L: 1 verified. Result: OK
I20240430 09:43:43.616317 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470207765683 usec, L: 1 Expected Rows: 101650
I20240430 09:43:43.629159 20885 meta_cache.cc:1510] marking tablet server 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) as failed
W20240430 09:43:43.629475 20885 meta_cache.cc:302] tablet f9388a35737a46d2bd003dddb7efe1f0: replica 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:43.642458 20885 linked_list-test-util.h:528] Calling callback on tserver 8d65d5b97d40479c8d0d65c30dda3557
I20240430 09:43:44.170689 22614 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3719f20ee14a02a13b62ef7d239d53)
I20240430 09:43:44.171038 22614 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:44.177911 22614 leader_election.cc:288] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427), 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963)
W20240430 09:43:44.189018 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427): Network error: Client connection negotiation failed: client connection to 127.20.101.67:36427: connect: Connection refused (error 111)
W20240430 09:43:44.189267 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963): Network error: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:44.189462 22470 leader_election.cc:302] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d65d5b97d40479c8d0d65c30dda3557; no voters: 2f3719f20ee14a02a13b62ef7d239d53, 68003d097d54450cb9ddf0d0c753eb0e
I20240430 09:43:44.189741 22614 raft_consensus.cc:2726] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:43:45.782495 20885 linked_list-test-util.h:767] Done collecting results (101650 rows in 2140.19ms)
I20240430 09:43:45.906275 20885 linked_list-test-util.h:556] Snapshot: P: 1714470207765683 usec, L: 1 verified. Result: OK
I20240430 09:43:45.909397 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 1714470217770125 usec, L: 1 Expected Rows: 151300
I20240430 09:43:45.910841 20885 meta_cache.cc:1510] marking tablet server 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) as failed
W20240430 09:43:45.910952 20885 meta_cache.cc:302] tablet f9388a35737a46d2bd003dddb7efe1f0: replica 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:45.919577 20885 linked_list-test-util.h:528] Calling callback on tserver 8d65d5b97d40479c8d0d65c30dda3557
I20240430 09:43:47.406930 22619 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3719f20ee14a02a13b62ef7d239d53)
I20240430 09:43:47.407110 22619 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:47.407737 22619 leader_election.cc:288] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427), 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963)
W20240430 09:43:47.408977 22470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.101.67:36427: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240430 09:43:47.410640 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963): Network error: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
W20240430 09:43:47.410830 22470 leader_election.cc:334] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68003d097d54450cb9ddf0d0c753eb0e (127.20.101.67:36427): Network error: Client connection negotiation failed: client connection to 127.20.101.67:36427: connect: Connection refused (error 111)
I20240430 09:43:47.410919 22470 leader_election.cc:302] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d65d5b97d40479c8d0d65c30dda3557; no voters: 2f3719f20ee14a02a13b62ef7d239d53, 68003d097d54450cb9ddf0d0c753eb0e
I20240430 09:43:47.411137 22619 raft_consensus.cc:2726] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 09:43:48.033118 20885 linked_list-test-util.h:767] Done collecting results (151300 rows in 2113.64ms)
I20240430 09:43:48.240275 20885 linked_list-test-util.h:556] Snapshot: P: 1714470217770125 usec, L: 1 verified. Result: OK
I20240430 09:43:48.243600 20885 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 151300
I20240430 09:43:48.245172 20885 meta_cache.cc:1510] marking tablet server 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) as failed
W20240430 09:43:48.245298 20885 meta_cache.cc:302] tablet f9388a35737a46d2bd003dddb7efe1f0: replica 2f3719f20ee14a02a13b62ef7d239d53 (127.20.101.66:39963) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.101.66:39963: connect: Connection refused (error 111)
I20240430 09:43:48.249300 20885 linked_list-test-util.h:528] Calling callback on tserver 8d65d5b97d40479c8d0d65c30dda3557
I20240430 09:43:50.283478 20885 linked_list-test-util.h:767] Done collecting results (151300 rows in 2034.27ms)
I20240430 09:43:50.471141 20885 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240430 09:43:50.472015 20885 linked_list-test-util.h:692] Successfully verified 151300 rows
I20240430 09:43:50.472973 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 22453
W20240430 09:43:50.472973 22223 client.cc:2025] Couldn't close scanner 6cace5e9ff524f5294ebf98cc1c267f3: Service unavailable: reactor is shutting down (error 108)
I20240430 09:43:50.480717 22579 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:50.716323 22627 raft_consensus.cc:482] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3719f20ee14a02a13b62ef7d239d53)
I20240430 09:43:50.716584 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 22453
I20240430 09:43:50.716511 22627 raft_consensus.cc:504] T f9388a35737a46d2bd003dddb7efe1f0 P 8d65d5b97d40479c8d0d65c30dda3557 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68003d097d54450cb9ddf0d0c753eb0e" member_type: VOTER last_known_addr { host: "127.20.101.67" port: 36427 } } peers { permanent_uuid: "8d65d5b97d40479c8d0d65c30dda3557" member_type: VOTER last_known_addr { host: "127.20.101.65" port: 41927 } } peers { permanent_uuid: "2f3719f20ee14a02a13b62ef7d239d53" member_type: VOTER last_known_addr { host: "127.20.101.66" port: 39963 } }
I20240430 09:43:50.771040 20885 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGww_u/build/asan/bin/kudu pid 21741
I20240430 09:43:50.772167 21802 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 09:43:50.880789 20885 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGww_u/build/asan/bin/kudu with pid 21741
2024-04-30T09:43:50Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (45686 ms)
[----------] 2 tests from LinkedListTest (83205 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (83205 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST