LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 69dc2bce073244fb8d08141f1c1a0bda, Rpc address: host: "127.2.46.66" port: 39777 (Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111))
I20240502 01:48:49.634397  2233 webui_checker.cc:88] shutting down CURL thread
W20240502 01:48:50.089680  2807 debug-util.cc:398] Leaking SignalData structure 0x7b08000d33e0 after lost signal to thread 2606
W20240502 01:48:50.194442  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:48:50.252285  2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 01:48:50.254915  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:48:51.762813  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:52.710737  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:52.836845  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:54.026682  2808 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4040 after lost signal to thread 2340
W20240502 01:48:54.027420  2808 debug-util.cc:398] Leaking SignalData structure 0x7b08000d47a0 after lost signal to thread 2467
W20240502 01:48:54.346639  2358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:48:54.425338  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:48:55.320326  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:48:55.361524  2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:48:55.366029  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:48:56.299474  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2339
I20240502 01:48:56.346400  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2605
I20240502 01:48:56.386099  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2247
2024-05-02T01:48:56Z chronyd exiting
I20240502 01:48:56.434173  2233 test_util.cc:175] -----------------------------------------------
I20240502 01:48:56.434327  2233 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:48:35.022989  2233 test_util.cc:268] Using random seed: -854049786
W20240502 01:48:36.061755  2233 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.997s	user 0.373s	sys 0.622s
W20240502 01:48:36.062070  2233 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.997s	user 0.373s	sys 0.622s
I20240502 01:48:36.072439  2233 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 01:48:36.072607  2233 linked_list-test.cc:101] --------------
I20240502 01:48:36.072770  2233 linked_list-test.cc:102] 50 chains
I20240502 01:48:36.072942  2233 linked_list-test.cc:103] 3 tablets
I20240502 01:48:36.073083  2233 linked_list-test.cc:104] Mutations on
I20240502 01:48:36.073235  2233 linked_list-test.cc:105] --------------
I20240502 01:48:36.073786  2233 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:48:36.073954  2233 ts_itest-base.cc:113] --------------
I20240502 01:48:36.074111  2233 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:48:36.074259  2233 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:48:36.074414  2233 ts_itest-base.cc:116] --------------
2024-05-02T01:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:48:36Z Disabled control of system clock
I20240502 01:48:36.109417  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.46.126:46359
--webserver_interface=127.2.46.126
--webserver_port=0
--builtin_ntp_servers=127.2.46.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.46.126:46359
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:48:36.375054  2247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:36.375516  2247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:36.375959  2247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:36.400681  2247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:48:36.400892  2247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:36.401124  2247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:48:36.401325  2247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:48:36.428759  2247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.46.126:46359
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.46.126:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.2.46.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:36.429654  2247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:36.430959  2247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:36.445545  2254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.445600  2256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.446476  2253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.421803  2247 server_base.cc:1034] running on GCE node
I20240502 01:48:37.423947  2247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:37.426365  2247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:37.427729  2247 hybrid_clock.cc:648] HybridClock initialized: now 1714614517427672 us; error 70 us; skew 500 ppm
I20240502 01:48:37.428503  2247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:37.434478  2247 webserver.cc:460] Webserver started at http://127.2.46.126:39999/ using document root <none> and password file <none>
I20240502 01:48:37.435323  2247 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:37.435519  2247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:37.435962  2247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:37.440634  2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/instance:
uuid: "bf556177853840db9e205509cd360bb8"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-hwlc"
I20240502 01:48:37.441560  2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal/instance:
uuid: "bf556177853840db9e205509cd360bb8"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-hwlc"
I20240502 01:48:37.447978  2247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:48:37.452739  2263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:37.453657  2247 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 01:48:37.453915  2247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
uuid: "bf556177853840db9e205509cd360bb8"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-hwlc"
I20240502 01:48:37.454201  2247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:37.501116  2247 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:37.502360  2247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:37.502756  2247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:37.569569  2247 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.126:46359
I20240502 01:48:37.569643  2314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.126:46359 every 8 connection(s)
I20240502 01:48:37.572049  2247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
I20240502 01:48:37.575837  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 2247
I20240502 01:48:37.576301  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal/instance
I20240502 01:48:37.577584  2315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:37.595209  2315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8: Bootstrap starting.
I20240502 01:48:37.599753  2315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:37.601835  2315 log.cc:824] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:37.606550  2315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8: No bootstrap required, opened a new log
I20240502 01:48:37.626509  2315 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } }
I20240502 01:48:37.627095  2315 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:37.627342  2315 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf556177853840db9e205509cd360bb8, State: Initialized, Role: FOLLOWER
I20240502 01:48:37.628036  2315 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } }
I20240502 01:48:37.628473  2315 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:37.628698  2315 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:37.628959  2315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:37.632553  2315 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } }
I20240502 01:48:37.633188  2315 leader_election.cc:302] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf556177853840db9e205509cd360bb8; no voters: 
I20240502 01:48:37.634750  2315 leader_election.cc:288] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:48:37.635432  2320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:37.637441  2320 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [term 1 LEADER]: Becoming Leader. State: Replica: bf556177853840db9e205509cd360bb8, State: Running, Role: LEADER
I20240502 01:48:37.638276  2320 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } }
I20240502 01:48:37.639854  2315 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:37.650229  2322 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf556177853840db9e205509cd360bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } } }
I20240502 01:48:37.650552  2321 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf556177853840db9e205509cd360bb8. Latest consensus state: current_term: 1 leader_uuid: "bf556177853840db9e205509cd360bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf556177853840db9e205509cd360bb8" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 46359 } } }
I20240502 01:48:37.651109  2322 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:37.651228  2321 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:37.653582  2330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:37.664188  2330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:37.679790  2330 catalog_manager.cc:1347] Generated new cluster ID: 2c40f7dcaa53408e965c0ee0074ee716
I20240502 01:48:37.679996  2330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:37.702987  2330 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:37.704087  2330 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:37.720558  2330 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf556177853840db9e205509cd360bb8: Generated new TSK 0
I20240502 01:48:37.721227  2330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:37.728211  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.65:0
--local_ip_for_outbound_sockets=127.2.46.65
--webserver_interface=127.2.46.65
--webserver_port=0
--tserver_master_addrs=127.2.46.126:46359
--builtin_ntp_servers=127.2.46.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:48:38.024504  2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:38.024997  2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:38.025511  2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:38.053563  2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:38.054387  2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.65
I20240502 01:48:38.084370  2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.2.46.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:46359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.65
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:38.085456  2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:38.086936  2339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:38.103142  2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:38.105634  2339 server_base.cc:1034] running on GCE node
W20240502 01:48:38.105715  2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:38.104207  2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:39.123661  2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:39.126271  2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:39.127703  2339 hybrid_clock.cc:648] HybridClock initialized: now 1714614519127654 us; error 48 us; skew 500 ppm
I20240502 01:48:39.128720  2339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:39.135469  2339 webserver.cc:460] Webserver started at http://127.2.46.65:33473/ using document root <none> and password file <none>
I20240502 01:48:39.136528  2339 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:39.136796  2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:39.137363  2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:39.143959  2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/instance:
uuid: "b16644ccdf9b430f88938474d8ab3060"
format_stamp: "Formatted at 2024-05-02 01:48:39 on dist-test-slave-hwlc"
I20240502 01:48:39.145243  2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal/instance:
uuid: "b16644ccdf9b430f88938474d8ab3060"
format_stamp: "Formatted at 2024-05-02 01:48:39 on dist-test-slave-hwlc"
I20240502 01:48:39.153358  2339 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.005s
I20240502 01:48:39.159847  2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:39.160818  2339 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:48:39.161150  2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
uuid: "b16644ccdf9b430f88938474d8ab3060"
format_stamp: "Formatted at 2024-05-02 01:48:39 on dist-test-slave-hwlc"
I20240502 01:48:39.161504  2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:39.222877  2339 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:39.224483  2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:39.224908  2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:39.227543  2339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:39.231393  2339 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:39.231595  2339 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:39.231812  2339 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:39.231963  2339 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:39.365193  2339 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.65:33639
I20240502 01:48:39.365284  2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.65:33639 every 8 connection(s)
I20240502 01:48:39.367513  2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:48:39.368686  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 2339
I20240502 01:48:39.369060  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal/instance
I20240502 01:48:39.374871  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.66:0
--local_ip_for_outbound_sockets=127.2.46.66
--webserver_interface=127.2.46.66
--webserver_port=0
--tserver_master_addrs=127.2.46.126:46359
--builtin_ntp_servers=127.2.46.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
I20240502 01:48:39.389735  2468 heartbeater.cc:347] Connected to a master server at 127.2.46.126:46359
I20240502 01:48:39.390085  2468 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:39.390987  2468 heartbeater.cc:511] Master 127.2.46.126:46359 requested a full tablet report, sending...
I20240502 01:48:39.393150  2280 ts_manager.cc:194] Registered new tserver with Master: b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639)
I20240502 01:48:39.394816  2280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.65:60523
W20240502 01:48:39.653443  2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:39.653954  2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:39.654474  2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:39.681608  2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:39.682431  2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.66
I20240502 01:48:39.713191  2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.2.46.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:46359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.66
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:39.714236  2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:39.715687  2472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:39.730630  2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.397754  2468 heartbeater.cc:503] Master 127.2.46.126:46359 was elected leader, sending a full tablet report...
W20240502 01:48:39.730664  2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:39.732915  2472 server_base.cc:1034] running on GCE node
W20240502 01:48:39.731926  2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.786734  2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:40.789499  2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:40.790902  2472 hybrid_clock.cc:648] HybridClock initialized: now 1714614520790866 us; error 48 us; skew 500 ppm
I20240502 01:48:40.791601  2472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:40.798242  2472 webserver.cc:460] Webserver started at http://127.2.46.66:38829/ using document root <none> and password file <none>
I20240502 01:48:40.799217  2472 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:40.799407  2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:40.799960  2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:40.804507  2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/instance:
uuid: "69dc2bce073244fb8d08141f1c1a0bda"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-hwlc"
I20240502 01:48:40.805541  2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal/instance:
uuid: "69dc2bce073244fb8d08141f1c1a0bda"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-hwlc"
I20240502 01:48:40.812350  2472 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 01:48:40.817255  2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:40.818224  2472 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 01:48:40.818519  2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
uuid: "69dc2bce073244fb8d08141f1c1a0bda"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-hwlc"
I20240502 01:48:40.818827  2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:40.866159  2472 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:40.867483  2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:40.867933  2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:40.870282  2472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:40.874341  2472 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:40.874533  2472 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:40.874750  2472 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:40.874905  2472 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:41.007536  2472 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.66:39777
I20240502 01:48:41.007647  2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.66:39777 every 8 connection(s)
I20240502 01:48:41.009925  2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
I20240502 01:48:41.014418  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 2472
I20240502 01:48:41.014894  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal/instance
I20240502 01:48:41.021286  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.67:0
--local_ip_for_outbound_sockets=127.2.46.67
--webserver_interface=127.2.46.67
--webserver_port=0
--tserver_master_addrs=127.2.46.126:46359
--builtin_ntp_servers=127.2.46.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
I20240502 01:48:41.032946  2601 heartbeater.cc:347] Connected to a master server at 127.2.46.126:46359
I20240502 01:48:41.033325  2601 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:41.034299  2601 heartbeater.cc:511] Master 127.2.46.126:46359 requested a full tablet report, sending...
I20240502 01:48:41.036336  2280 ts_manager.cc:194] Registered new tserver with Master: 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
I20240502 01:48:41.038226  2280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.66:54841
W20240502 01:48:41.305560  2605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:41.305958  2605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:41.306397  2605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:41.333503  2605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:41.334271  2605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.67
I20240502 01:48:41.364599  2605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.2.46.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:46359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.67
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:41.365640  2605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:41.367098  2605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:41.382089  2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:42.041414  2601 heartbeater.cc:503] Master 127.2.46.126:46359 was elected leader, sending a full tablet report...
W20240502 01:48:41.382138  2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:41.382270  2611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:42.392365  2613 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 01:48:42.392469  2605 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 01:48:42.396132  2605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:42.398319  2605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:42.399665  2605 hybrid_clock.cc:648] HybridClock initialized: now 1714614522399627 us; error 43 us; skew 500 ppm
I20240502 01:48:42.400405  2605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:42.405999  2605 webserver.cc:460] Webserver started at http://127.2.46.67:36355/ using document root <none> and password file <none>
I20240502 01:48:42.407052  2605 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:42.407275  2605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:42.407708  2605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:42.412396  2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/instance:
uuid: "5ad6bd000714465092a095414dba4338"
format_stamp: "Formatted at 2024-05-02 01:48:42 on dist-test-slave-hwlc"
I20240502 01:48:42.413376  2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal/instance:
uuid: "5ad6bd000714465092a095414dba4338"
format_stamp: "Formatted at 2024-05-02 01:48:42 on dist-test-slave-hwlc"
I20240502 01:48:42.419723  2605 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:48:42.424515  2621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:42.425366  2605 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20240502 01:48:42.425621  2605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
uuid: "5ad6bd000714465092a095414dba4338"
format_stamp: "Formatted at 2024-05-02 01:48:42 on dist-test-slave-hwlc"
I20240502 01:48:42.425926  2605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:42.481271  2605 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:42.482611  2605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:42.483001  2605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:42.485319  2605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:42.489009  2605 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:42.489188  2605 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:42.489403  2605 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:42.489540  2605 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:42.619316  2605 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.67:41975
I20240502 01:48:42.619426  2733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.67:41975 every 8 connection(s)
I20240502 01:48:42.621701  2605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
I20240502 01:48:42.630424  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 2605
I20240502 01:48:42.630753  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal/instance
I20240502 01:48:42.640869  2734 heartbeater.cc:347] Connected to a master server at 127.2.46.126:46359
I20240502 01:48:42.641203  2734 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:42.641978  2734 heartbeater.cc:511] Master 127.2.46.126:46359 requested a full tablet report, sending...
I20240502 01:48:42.643673  2280 ts_manager.cc:194] Registered new tserver with Master: 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:42.644847  2280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.67:42407
I20240502 01:48:42.649506  2233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:48:42.681056  2280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47316:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240502 01:48:42.706629  2280 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:48:42.759132  2401 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdb154fb4b947e7aa46a952036daa40 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:48:42.761193  2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdb154fb4b947e7aa46a952036daa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.761592  2402 tablet_service.cc:1467] Processing CreateTablet for tablet f6dc4786dff143caa9dee619976264e5 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:48:42.762589  2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dc4786dff143caa9dee619976264e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.763381  2403 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe288ea889a4b3380330374e0568835 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:48:42.764429  2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe288ea889a4b3380330374e0568835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.773432  2534 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdb154fb4b947e7aa46a952036daa40 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:48:42.775363  2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdb154fb4b947e7aa46a952036daa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.774973  2535 tablet_service.cc:1467] Processing CreateTablet for tablet f6dc4786dff143caa9dee619976264e5 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:48:42.776158  2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dc4786dff143caa9dee619976264e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.776589  2668 tablet_service.cc:1467] Processing CreateTablet for tablet f6dc4786dff143caa9dee619976264e5 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:48:42.778266  2536 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe288ea889a4b3380330374e0568835 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:48:42.778827  2668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dc4786dff143caa9dee619976264e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.776933  2669 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe288ea889a4b3380330374e0568835 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:48:42.784873  2669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe288ea889a4b3380330374e0568835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.776592  2667 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdb154fb4b947e7aa46a952036daa40 (DEFAULT_TABLE table=TestTable [id=0cc6bb8453c74fd29ec0241657fa0ac0]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:48:42.786968  2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdb154fb4b947e7aa46a952036daa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.779464  2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe288ea889a4b3380330374e0568835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.822814  2753 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: Bootstrap starting.
I20240502 01:48:42.834805  2755 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Bootstrap starting.
I20240502 01:48:42.836707  2753 tablet_bootstrap.cc:654] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.838510  2754 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: Bootstrap starting.
I20240502 01:48:42.838928  2753 log.cc:824] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:42.843572  2755 tablet_bootstrap.cc:654] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.843708  2754 tablet_bootstrap.cc:654] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.844400  2753 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: No bootstrap required, opened a new log
I20240502 01:48:42.844862  2753 ts_tablet_manager.cc:1367] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.002s
I20240502 01:48:42.845304  2754 log.cc:824] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:42.845608  2755 log.cc:824] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:42.850250  2755 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: No bootstrap required, opened a new log
I20240502 01:48:42.850328  2754 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: No bootstrap required, opened a new log
I20240502 01:48:42.850628  2755 ts_tablet_manager.cc:1367] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.004s
I20240502 01:48:42.850740  2754 ts_tablet_manager.cc:1367] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.005s
I20240502 01:48:42.869248  2755 raft_consensus.cc:348] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.869792  2755 raft_consensus.cc:374] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.870077  2755 raft_consensus.cc:729] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad6bd000714465092a095414dba4338, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.870867  2755 consensus_queue.cc:260] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.872426  2753 raft_consensus.cc:348] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.873276  2753 raft_consensus.cc:374] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.873600  2734 heartbeater.cc:503] Master 127.2.46.126:46359 was elected leader, sending a full tablet report...
I20240502 01:48:42.873555  2753 raft_consensus.cc:729] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16644ccdf9b430f88938474d8ab3060, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.874305  2755 ts_tablet_manager.cc:1398] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Time spent starting tablet: real 0.023s	user 0.020s	sys 0.003s
I20240502 01:48:42.874337  2753 consensus_queue.cc:260] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.875087  2755 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338: Bootstrap starting.
W20240502 01:48:42.876001  2735 tablet.cc:2369] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:48:42.880267  2755 tablet_bootstrap.cc:654] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.880895  2754 raft_consensus.cc:348] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.881878  2754 raft_consensus.cc:374] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.883806  2753 ts_tablet_manager.cc:1398] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060: Time spent starting tablet: real 0.039s	user 0.029s	sys 0.010s
I20240502 01:48:42.882182  2754 raft_consensus.cc:729] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dc2bce073244fb8d08141f1c1a0bda, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.884544  2753 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: Bootstrap starting.
I20240502 01:48:42.884943  2754 consensus_queue.cc:260] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.890424  2753 tablet_bootstrap.cc:654] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.896965  2754 ts_tablet_manager.cc:1398] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent starting tablet: real 0.046s	user 0.025s	sys 0.015s
I20240502 01:48:42.901062  2754 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda: Bootstrap starting.
I20240502 01:48:42.908689  2754 tablet_bootstrap.cc:654] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.910647  2753 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: No bootstrap required, opened a new log
I20240502 01:48:42.911554  2753 ts_tablet_manager.cc:1367] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.012s
I20240502 01:48:42.914480  2753 raft_consensus.cc:348] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.915339  2753 raft_consensus.cc:374] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.915693  2753 raft_consensus.cc:729] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16644ccdf9b430f88938474d8ab3060, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.917309  2753 consensus_queue.cc:260] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.918815  2755 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338: No bootstrap required, opened a new log
I20240502 01:48:42.919217  2755 ts_tablet_manager.cc:1367] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338: Time spent bootstrapping tablet: real 0.044s	user 0.023s	sys 0.020s
I20240502 01:48:42.919914  2754 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda: No bootstrap required, opened a new log
I20240502 01:48:42.920204  2754 ts_tablet_manager.cc:1367] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.003s
I20240502 01:48:42.921057  2753 ts_tablet_manager.cc:1398] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.002s
I20240502 01:48:42.921802  2753 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060: Bootstrap starting.
I20240502 01:48:42.921463  2755 raft_consensus.cc:348] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.922039  2755 raft_consensus.cc:374] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.922313  2755 raft_consensus.cc:729] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad6bd000714465092a095414dba4338, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.922518  2754 raft_consensus.cc:348] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.923183  2754 raft_consensus.cc:374] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.923463  2754 raft_consensus.cc:729] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dc2bce073244fb8d08141f1c1a0bda, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.923226  2755 consensus_queue.cc:260] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.924196  2754 consensus_queue.cc:260] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.926026  2754 ts_tablet_manager.cc:1398] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:48:42.926625  2753 tablet_bootstrap.cc:654] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.928586  2754 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: Bootstrap starting.
I20240502 01:48:42.930544  2755 ts_tablet_manager.cc:1398] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.003s
I20240502 01:48:42.934051  2754 tablet_bootstrap.cc:654] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.935041  2755 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338: Bootstrap starting.
I20240502 01:48:42.940300  2754 tablet_bootstrap.cc:492] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: No bootstrap required, opened a new log
I20240502 01:48:42.940266  2753 tablet_bootstrap.cc:492] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060: No bootstrap required, opened a new log
I20240502 01:48:42.940593  2753 ts_tablet_manager.cc:1367] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.013s
I20240502 01:48:42.940629  2754 ts_tablet_manager.cc:1367] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20240502 01:48:42.940665  2755 tablet_bootstrap.cc:654] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.942376  2754 raft_consensus.cc:348] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.942780  2754 raft_consensus.cc:374] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.942970  2754 raft_consensus.cc:729] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dc2bce073244fb8d08141f1c1a0bda, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.942713  2753 raft_consensus.cc:348] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.943475  2753 raft_consensus.cc:374] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.943850  2753 raft_consensus.cc:729] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16644ccdf9b430f88938474d8ab3060, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.943864  2754 consensus_queue.cc:260] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:42.944325  2753 consensus_queue.cc:260] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.945629  2755 tablet_bootstrap.cc:492] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338: No bootstrap required, opened a new log
I20240502 01:48:42.945935  2754 ts_tablet_manager.cc:1398] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:48:42.945977  2755 ts_tablet_manager.cc:1367] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240502 01:48:42.946167  2753 ts_tablet_manager.cc:1398] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:48:42.948101  2755 raft_consensus.cc:348] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.948556  2755 raft_consensus.cc:374] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.948741  2755 raft_consensus.cc:729] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad6bd000714465092a095414dba4338, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.949177  2755 consensus_queue.cc:260] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.950610  2755 ts_tablet_manager.cc:1398] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:48:42.971622  2760 raft_consensus.cc:482] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.971983  2760 raft_consensus.cc:504] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.974009  2760 leader_election.cc:288] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:42.984182  2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dc4786dff143caa9dee619976264e5" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dc2bce073244fb8d08141f1c1a0bda" is_pre_election: true
I20240502 01:48:42.985083  2556 raft_consensus.cc:2454] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 0.
I20240502 01:48:42.985653  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dc4786dff143caa9dee619976264e5" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338" is_pre_election: true
I20240502 01:48:42.986166  2358 leader_election.cc:302] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:42.986325  2689 raft_consensus.cc:2454] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 0.
I20240502 01:48:42.986742  2760 raft_consensus.cc:2781] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.986984  2760 raft_consensus.cc:482] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.987308  2760 raft_consensus.cc:3037] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.990957  2760 raft_consensus.cc:504] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:42.992159  2760 leader_election.cc:288] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 1 election: Requested vote from peers 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:42.992681  2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dc4786dff143caa9dee619976264e5" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dc2bce073244fb8d08141f1c1a0bda"
I20240502 01:48:42.993001  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dc4786dff143caa9dee619976264e5" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338"
I20240502 01:48:42.993177  2556 raft_consensus.cc:3037] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.993464  2689 raft_consensus.cc:3037] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.999287  2556 raft_consensus.cc:2454] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 1.
I20240502 01:48:42.999502  2689 raft_consensus.cc:2454] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 1.
I20240502 01:48:43.000142  2358 leader_election.cc:302] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:43.000617  2760 raft_consensus.cc:2781] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:43.001930  2760 raft_consensus.cc:686] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [term 1 LEADER]: Becoming Leader. State: Replica: b16644ccdf9b430f88938474d8ab3060, State: Running, Role: LEADER
I20240502 01:48:43.002557  2760 consensus_queue.cc:237] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:43.012856  2279 catalog_manager.cc:5484] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 reported cstate change: term changed from 0 to 1, leader changed from <none> to b16644ccdf9b430f88938474d8ab3060 (127.2.46.65). New cstate: current_term: 1 leader_uuid: "b16644ccdf9b430f88938474d8ab3060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } health_report { overall_health: UNKNOWN } } }
W20240502 01:48:43.017413  2602 tablet.cc:2369] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:48:43.035989  2762 raft_consensus.cc:482] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:43.036300  2762 raft_consensus.cc:504] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:43.038893  2762 leader_election.cc:288] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:43.050277  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060" is_pre_election: true
I20240502 01:48:43.050828  2423 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 0.
I20240502 01:48:43.051293  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338" is_pre_election: true
I20240502 01:48:43.051755  2689 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 0.
I20240502 01:48:43.051858  2490 leader_election.cc:302] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:43.052511  2762 raft_consensus.cc:2781] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:43.052714  2762 raft_consensus.cc:482] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:43.052886  2762 raft_consensus.cc:3037] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.056775  2762 raft_consensus.cc:504] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:43.058318  2762 leader_election.cc:288] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 election: Requested vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:43.058923  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060"
I20240502 01:48:43.059044  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338"
I20240502 01:48:43.059255  2423 raft_consensus.cc:3037] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.059407  2689 raft_consensus.cc:3037] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.062588  2689 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 1.
I20240502 01:48:43.062873  2423 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 1.
I20240502 01:48:43.063452  2491 leader_election.cc:302] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad6bd000714465092a095414dba4338, 69dc2bce073244fb8d08141f1c1a0bda; no voters: 
I20240502 01:48:43.064169  2762 raft_consensus.cc:2781] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:43.065460  2762 raft_consensus.cc:686] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 LEADER]: Becoming Leader. State: Replica: 69dc2bce073244fb8d08141f1c1a0bda, State: Running, Role: LEADER
I20240502 01:48:43.066041  2762 consensus_queue.cc:237] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:43.075659  2279 catalog_manager.cc:5484] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66). New cstate: current_term: 1 leader_uuid: "69dc2bce073244fb8d08141f1c1a0bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:43.089864  2762 raft_consensus.cc:482] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:43.090134  2762 raft_consensus.cc:504] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:43.091193  2762 leader_election.cc:288] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:43.091816  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060" is_pre_election: true
I20240502 01:48:43.091925  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338" is_pre_election: true
I20240502 01:48:43.092234  2423 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 0.
I20240502 01:48:43.092350  2689 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 0.
I20240502 01:48:43.092896  2490 leader_election.cc:302] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:43.093389  2762 raft_consensus.cc:2781] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:43.093644  2762 raft_consensus.cc:482] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:43.093848  2762 raft_consensus.cc:3037] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.096997  2762 raft_consensus.cc:504] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:43.098416  2762 leader_election.cc:288] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 election: Requested vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
I20240502 01:48:43.098874  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060"
I20240502 01:48:43.099241  2423 raft_consensus.cc:3037] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.099160  2689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "69dc2bce073244fb8d08141f1c1a0bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338"
I20240502 01:48:43.099494  2689 raft_consensus.cc:3037] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:43.102336  2423 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 1.
I20240502 01:48:43.102546  2689 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69dc2bce073244fb8d08141f1c1a0bda in term 1.
I20240502 01:48:43.102998  2490 leader_election.cc:302] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:43.103500  2762 raft_consensus.cc:2781] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:43.103868  2762 raft_consensus.cc:686] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 LEADER]: Becoming Leader. State: Replica: 69dc2bce073244fb8d08141f1c1a0bda, State: Running, Role: LEADER
I20240502 01:48:43.104411  2762 consensus_queue.cc:237] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:43.110427  2279 catalog_manager.cc:5484] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66). New cstate: current_term: 1 leader_uuid: "69dc2bce073244fb8d08141f1c1a0bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } health_report { overall_health: HEALTHY } } }
W20240502 01:48:43.125918  2469 tablet.cc:2369] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:48:43.132179  2233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:48:43.135645  2233 ts_itest-base.cc:233] Waiting for 3 tablets on tserver b16644ccdf9b430f88938474d8ab3060 to finish bootstrapping
I20240502 01:48:43.153234  2233 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 69dc2bce073244fb8d08141f1c1a0bda to finish bootstrapping
I20240502 01:48:43.163229  2233 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5ad6bd000714465092a095414dba4338 to finish bootstrapping
I20240502 01:48:43.175221  2779 webui_checker.cc:102] curl thread will poll the following URLs every 0.010s:
  http://127.2.46.126:39999/ipki-ca-cert
  http://127.2.46.66:38829/mem-trackers
  http://127.2.46.66:38829/metrics
  http://127.2.46.67:36355/mem-trackers
  http://127.2.46.65:33473/scans
  http://127.2.46.66:38829/threadz
  http://127.2.46.66:38829/transactions?tablet_id=8cdb154fb4b947e7aa46a952036daa40
  http://127.2.46.126:39999/dump-entities
  http://127.2.46.65:33473/metrics
  http://127.2.46.126:39999/ipki-ca-cert-der
  http://127.2.46.126:39999/stacks
  http://127.2.46.66:38829/startup
  http://127.2.46.66:38829/scans
  http://127.2.46.65:33473/startup
  http://127.2.46.67:36355/metrics
  http://127.2.46.126:39999/ipki-ca-cert-pem
  http://127.2.46.67:36355/startup
  http://127.2.46.65:33473/transactions?tablet_id=8cdb154fb4b947e7aa46a952036daa40
  http://127.2.46.65:33473/threadz?group=all
  http://127.2.46.126:39999/mem-trackers
  http://127.2.46.126:39999/masters
  http://127.2.46.126:39999/threadz
  http://127.2.46.65:33473/mem-trackers
  http://127.2.46.66:38829/maintenance-manager
  http://127.2.46.126:39999/threadz?group=all
  http://127.2.46.67:36355/threadz?group=all
  http://127.2.46.126:39999/tables
  http://127.2.46.67:36355/transactions?tablet_id=8cdb154fb4b947e7aa46a952036daa40
  http://127.2.46.67:36355/maintenance-manager
  http://127.2.46.67:36355/scans
  http://127.2.46.67:36355/threadz
  http://127.2.46.66:38829/threadz?group=all
  http://127.2.46.66:38829/tablets
  http://127.2.46.67:36355/stacks
  http://127.2.46.126:39999/tablet-servers
  http://127.2.46.65:33473/threadz
  http://127.2.46.65:33473/stacks
  http://127.2.46.66:38829/stacks
  http://127.2.46.67:36355/tablets
  http://127.2.46.65:33473/tablets
  http://127.2.46.126:39999/metrics
  http://127.2.46.65:33473/maintenance-manager
  http://127.2.46.126:39999/startup
I20240502 01:48:43.181192  2233 linked_list-test-util.h:401] Running for: 5.000s
I20240502 01:48:43.181383  2233 linked_list-test-util.h:402] Sampling every 1666666 us
I20240502 01:48:43.181540  2233 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240502 01:48:43.245075  2689 raft_consensus.cc:1261] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Refusing update from remote peer 69dc2bce073244fb8d08141f1c1a0bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.247961  2774 consensus_queue.cc:1035] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.252079  2423 raft_consensus.cc:1261] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Refusing update from remote peer 69dc2bce073244fb8d08141f1c1a0bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.252173  2556 raft_consensus.cc:1261] T f6dc4786dff143caa9dee619976264e5 P 69dc2bce073244fb8d08141f1c1a0bda [term 1 FOLLOWER]: Refusing update from remote peer b16644ccdf9b430f88938474d8ab3060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.253654  2760 consensus_queue.cc:1035] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.260977  2689 raft_consensus.cc:1261] T f6dc4786dff143caa9dee619976264e5 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Refusing update from remote peer b16644ccdf9b430f88938474d8ab3060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.263181  2760 consensus_queue.cc:1035] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.264815  2790 consensus_queue.cc:1035] T 3fe288ea889a4b3380330374e0568835 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.288637  2689 raft_consensus.cc:1261] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Refusing update from remote peer 69dc2bce073244fb8d08141f1c1a0bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.288775  2422 raft_consensus.cc:1261] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Refusing update from remote peer 69dc2bce073244fb8d08141f1c1a0bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:43.294281  2762 consensus_queue.cc:1035] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.298080  2774 consensus_queue.cc:1035] T 8cdb154fb4b947e7aa46a952036daa40 P 69dc2bce073244fb8d08141f1c1a0bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.345593  2796 mvcc.cc:205] Tried to move back new op lower bound from 7023061087213625344 to 7023061086224646144. Current Snapshot: MvccSnapshot[applied={T|T < 7023061087213625344}]
I20240502 01:48:43.352231  2788 mvcc.cc:205] Tried to move back new op lower bound from 7023061087213621248 to 7023061086484553728. Current Snapshot: MvccSnapshot[applied={T|T < 7023061087213621248}]
I20240502 01:48:43.353241  2793 mvcc.cc:205] Tried to move back new op lower bound from 7023061087213621248 to 7023061086484553728. Current Snapshot: MvccSnapshot[applied={T|T < 7023061087213621248}]
I20240502 01:48:43.481181  2808 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 01:48:43.514487  2782 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7eff20e3d950  google::LogMessage::Flush() at ??:0
    @     0x7eff20e42a8b  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7eff22fffc3c  kudu::operator-() at ??:0
    @     0x7eff2a45403c  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7eff2a43cc4e  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7eff2a43cb92  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7eff2a403181  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7eff2b04b86a  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7eff2b054322  kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7eff2b0542c9  _ZNSt3__18__invokeIRZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS1_9WebserverEE3$_3JRKNS1_19WebCallbackRegistry10WebRequestEPNS8_22PrerenderedWebResponseEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSE_DpOSF_ at ??:0
    @     0x7eff2b05422d  std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
    @     0x7eff2b0541c9  std::__1::__function::__alloc_func<>::operator()() at ??:0
    @     0x7eff2b053491  std::__1::__function::__func<>::operator()() at ??:0
    @     0x7eff292a1feb  std::__1::__function::__value_func<>::operator()() at ??:0
    @     0x7eff29298242  std::__1::function<>::operator()() at ??:0
    @     0x7eff292926a9  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7eff29290d6c  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7eff29290119  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7eff292c3548  handle_request at ??:0
    @     0x7eff292c13d8  worker_thread at ??:0
    @           0x44ffc0  __tsan_thread_start_func at ??:?
    @     0x7eff246db6db  start_thread at ??:0
    @     0x7eff1fae271f  clone at ??:0
W20240502 01:48:43.996004  2740 meta_cache.cc:302] tablet 3fe288ea889a4b3380330374e0568835: replica 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777) has failed: Network error: recv got EOF from 127.2.46.66:39777 (error 108)
W20240502 01:48:44.002825  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.025465  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.044633  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.056914  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.082621  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.090262  2358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
W20240502 01:48:44.092939  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:48:44.096729  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.131459  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.151178  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.197691  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.217984  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.264851  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.286069  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.338033  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.363219  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.411185  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.416277  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.418428  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.420311  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.430620  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.440800  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.447031  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.448999  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.470213  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.473671  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.485448  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.493934  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.511684  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.517091  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.532872  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.537484  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.545022  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.563482  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.568926  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.580246  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.617112  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.620687  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.641304  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.649699  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.653447  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.691478  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.715384  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.718868  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.720512  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.759157  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.769544  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.796274  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.840719  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.855985  2383 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.857563  2382 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.880481  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.883862  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.887498  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.949537  2382 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
W20240502 01:48:44.967849  2382 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
I20240502 01:48:44.973997  2837 raft_consensus.cc:482] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:44.974298  2837 raft_consensus.cc:504] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:44.975523  2848 raft_consensus.cc:482] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:44.975637  2837 leader_election.cc:288] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975), 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
I20240502 01:48:44.975934  2848 raft_consensus.cc:504] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:44.976312  2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338" is_pre_election: true
I20240502 01:48:44.976910  2688 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 1.
I20240502 01:48:44.978566  2848 leader_election.cc:288] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
I20240502 01:48:44.978561  2358 leader_election.cc:302] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad6bd000714465092a095414dba4338, b16644ccdf9b430f88938474d8ab3060; no voters: 
W20240502 01:48:44.979204  2358 leader_election.cc:334] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:44.979925  2837 raft_consensus.cc:2781] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:48:44.980314  2837 raft_consensus.cc:482] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:44.980686  2837 raft_consensus.cc:3037] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:44.988163  2837 raft_consensus.cc:504] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
I20240502 01:48:44.990605  2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338"
I20240502 01:48:44.991101  2688 raft_consensus.cc:3037] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:44.991080  2837 leader_election.cc:288] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 election: Requested vote from peers 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975), 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
W20240502 01:48:44.994658  2358 leader_election.cc:334] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
W20240502 01:48:44.997535  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
I20240502 01:48:44.997982  2688 raft_consensus.cc:2454] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b16644ccdf9b430f88938474d8ab3060 in term 2.
I20240502 01:48:44.998855  2358 leader_election.cc:302] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ad6bd000714465092a095414dba4338, b16644ccdf9b430f88938474d8ab3060; no voters: 69dc2bce073244fb8d08141f1c1a0bda
I20240502 01:48:44.999523  2837 raft_consensus.cc:2781] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:44.999981  2837 raft_consensus.cc:686] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [term 2 LEADER]: Becoming Leader. State: Replica: b16644ccdf9b430f88938474d8ab3060, State: Running, Role: LEADER
I20240502 01:48:45.000655  2837 consensus_queue.cc:237] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } }
W20240502 01:48:45.001684  2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
W20240502 01:48:45.005218  2624 leader_election.cc:334] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:45.007089  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe288ea889a4b3380330374e0568835" candidate_uuid: "5ad6bd000714465092a095414dba4338" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060" is_pre_election: true
I20240502 01:48:45.008630  2623 leader_election.cc:302] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ad6bd000714465092a095414dba4338; no voters: 69dc2bce073244fb8d08141f1c1a0bda, b16644ccdf9b430f88938474d8ab3060
I20240502 01:48:45.008095  2277 catalog_manager.cc:5484] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 reported cstate change: term changed from 1 to 2, leader changed from 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66) to b16644ccdf9b430f88938474d8ab3060 (127.2.46.65). New cstate: current_term: 2 leader_uuid: "b16644ccdf9b430f88938474d8ab3060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:45.009629  2848 raft_consensus.cc:2726] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:48:45.011157  2649 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58870: Illegal state: replica 5ad6bd000714465092a095414dba4338 is not leader of this config: current role FOLLOWER
W20240502 01:48:45.023047  2382 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54606: Illegal state: replica b16644ccdf9b430f88938474d8ab3060 is not leader of this config: current role FOLLOWER
I20240502 01:48:45.039393  2848 raft_consensus.cc:482] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:45.039686  2848 raft_consensus.cc:504] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:45.041242  2848 leader_election.cc:288] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
I20240502 01:48:45.041389  2837 raft_consensus.cc:482] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:45.041934  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "5ad6bd000714465092a095414dba4338" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060" is_pre_election: true
I20240502 01:48:45.041801  2837 raft_consensus.cc:504] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:45.042652  2423 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad6bd000714465092a095414dba4338 in term 1.
I20240502 01:48:45.043545  2623 leader_election.cc:302] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad6bd000714465092a095414dba4338, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:45.044256  2848 raft_consensus.cc:2781] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:48:45.044538  2848 raft_consensus.cc:482] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69dc2bce073244fb8d08141f1c1a0bda)
I20240502 01:48:45.044836  2848 raft_consensus.cc:3037] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:45.045459  2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "b16644ccdf9b430f88938474d8ab3060" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ad6bd000714465092a095414dba4338" is_pre_election: true
I20240502 01:48:45.046058  2837 leader_election.cc:288] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777), 5ad6bd000714465092a095414dba4338 (127.2.46.67:41975)
W20240502 01:48:45.046854  2358 leader_election.cc:334] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
W20240502 01:48:45.049806  2624 leader_election.cc:334] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:45.051708  2848 raft_consensus.cc:504] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:45.052841  2688 raft_consensus.cc:2379] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b16644ccdf9b430f88938474d8ab3060 in current term 2: Already voted for candidate 5ad6bd000714465092a095414dba4338 in this term.
I20240502 01:48:45.053443  2687 raft_consensus.cc:1261] T 3fe288ea889a4b3380330374e0568835 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Refusing update from remote peer b16644ccdf9b430f88938474d8ab3060: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:48:45.054168  2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdb154fb4b947e7aa46a952036daa40" candidate_uuid: "5ad6bd000714465092a095414dba4338" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b16644ccdf9b430f88938474d8ab3060"
I20240502 01:48:45.054675  2423 raft_consensus.cc:3037] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:45.054736  2837 consensus_queue.cc:1035] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:48:45.055894  2358 leader_election.cc:302] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b16644ccdf9b430f88938474d8ab3060; no voters: 5ad6bd000714465092a095414dba4338, 69dc2bce073244fb8d08141f1c1a0bda
W20240502 01:48:45.056782  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:48:45.059032  2740 meta_cache.cc:302] tablet 8cdb154fb4b947e7aa46a952036daa40: replica 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777) has failed: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:45.060191  2423 raft_consensus.cc:2454] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad6bd000714465092a095414dba4338 in term 2.
I20240502 01:48:45.060194  2848 leader_election.cc:288] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 election: Requested vote from peers b16644ccdf9b430f88938474d8ab3060 (127.2.46.65:33639), 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777)
I20240502 01:48:45.060585  2845 raft_consensus.cc:2726] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:48:45.063437  2623 leader_election.cc:302] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad6bd000714465092a095414dba4338, b16644ccdf9b430f88938474d8ab3060; no voters: 
I20240502 01:48:45.064165  2848 raft_consensus.cc:2781] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 01:48:45.064699  2624 leader_election.cc:334] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:45.066882  2848 raft_consensus.cc:686] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [term 2 LEADER]: Becoming Leader. State: Replica: 5ad6bd000714465092a095414dba4338, State: Running, Role: LEADER
I20240502 01:48:45.067785  2848 consensus_queue.cc:237] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } }
I20240502 01:48:45.083918  2277 catalog_manager.cc:5484] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 reported cstate change: term changed from 1 to 2, leader changed from 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66) to 5ad6bd000714465092a095414dba4338 (127.2.46.67). New cstate: current_term: 2 leader_uuid: "5ad6bd000714465092a095414dba4338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69dc2bce073244fb8d08141f1c1a0bda" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad6bd000714465092a095414dba4338" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:45.101848  2423 raft_consensus.cc:1261] T 8cdb154fb4b947e7aa46a952036daa40 P b16644ccdf9b430f88938474d8ab3060 [term 2 FOLLOWER]: Refusing update from remote peer 5ad6bd000714465092a095414dba4338: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:48:45.103335  2862 consensus_queue.cc:1035] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b16644ccdf9b430f88938474d8ab3060" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:48:45.103946  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:45.138897  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061094862135296 Inserted count: 150
W20240502 01:48:45.169973  2780 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3700 after lost signal to thread 2248
W20240502 01:48:45.170691  2780 debug-util.cc:398] Leaking SignalData structure 0x7b080009d1e0 after lost signal to thread 2314
W20240502 01:48:46.398665  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:48:46.520231  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061100532744192 Inserted count: 1500
W20240502 01:48:47.426254  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:48:47.646040  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:48:48.199342  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061107435053056 Inserted count: 3100
I20240502 01:48:48.201858  2233 linked_list-test-util.h:421] Finished inserting list. Added 3100 in chain
I20240502 01:48:48.202096  2233 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 01:48:48.202286  2233 linked_list-test-util.h:424] 0: 5366660212558659584
I20240502 01:48:48.202472  2233 linked_list-test-util.h:424] 1: -4034556594667585535
I20240502 01:48:48.202704  2233 linked_list-test-util.h:424] 2: 5011111409304141826
I20240502 01:48:48.202878  2233 linked_list-test-util.h:424] 3: -4389823922945392637
I20240502 01:48:48.203050  2233 linked_list-test-util.h:424] 4: -8424239780124557308
I20240502 01:48:48.203202  2233 linked_list-test-util.h:424] 5: 621568961335459845
I20240502 01:48:48.203346  2233 linked_list-test-util.h:424] 6: -8779366370914074618
I20240502 01:48:48.203493  2233 linked_list-test-util.h:424] 7: 266301633057652743
I20240502 01:48:48.203636  2233 linked_list-test-util.h:424] 8: -3768254961609932792
I20240502 01:48:48.203810  2233 linked_list-test-util.h:424] 9: 5277413042361794569
I20240502 01:48:48.203982  2233 linked_list-test-util.h:424] 10: -4123522289887739894
I20240502 01:48:48.204123  2233 linked_list-test-util.h:424] 11: 4922286451572277259
I20240502 01:48:48.204257  2233 linked_list-test-util.h:424] 12: 887870594393112588
I20240502 01:48:48.204392  2233 linked_list-test-util.h:424] 13: -8513205475344711667
I20240502 01:48:48.204530  2233 linked_list-test-util.h:424] 14: 532603266115305486
I20240502 01:48:48.204689  2233 linked_list-test-util.h:424] 15: -8868332066134228977
I20240502 01:48:48.204836  2233 linked_list-test-util.h:424] 16: 5543714675419447312
I20240502 01:48:48.204981  2233 linked_list-test-util.h:424] 17: -3857220656830087151
I20240502 01:48:48.205127  2233 linked_list-test-util.h:424] 18: 5188447347141640210
I20240502 01:48:48.205269  2233 linked_list-test-util.h:424] 19: -4212487985107894253
I20240502 01:48:48.205412  2233 linked_list-test-util.h:424] 20: -8246903842287058924
I20240502 01:48:48.205556  2233 linked_list-test-util.h:424] 21: 798904899172958229
I20240502 01:48:48.205698  2233 linked_list-test-util.h:424] 22: -8602030433076576234
I20240502 01:48:48.205864  2233 linked_list-test-util.h:424] 23: 443637570895151127
I20240502 01:48:48.206008  2233 linked_list-test-util.h:424] 24: -3590919023772434408
I20240502 01:48:48.206156  2233 linked_list-test-util.h:424] 25: 5454748980199292953
I20240502 01:48:48.206296  2233 linked_list-test-util.h:424] 26: -3946186352050241510
I20240502 01:48:48.206436  2233 linked_list-test-util.h:424] 27: -7980602209229406181
I20240502 01:48:48.206580  2233 linked_list-test-util.h:424] 28: 1065206532230610972
I20240502 01:48:48.206719  2233 linked_list-test-util.h:424] 29: -8335869537507213283
I20240502 01:48:48.206876  2233 linked_list-test-util.h:424] 30: 709939203952803870
I20240502 01:48:48.207034  2233 linked_list-test-util.h:424] 31: -3324476653226360801
I20240502 01:48:48.207178  2233 linked_list-test-util.h:424] 32: 5721050613256945696
I20240502 01:48:48.207319  2233 linked_list-test-util.h:424] 33: -3679884718992588767
I20240502 01:48:48.207479  2233 linked_list-test-util.h:424] 34: 5365783284979138594
I20240502 01:48:48.207653  2233 linked_list-test-util.h:424] 35: 1331508165288263715
I20240502 01:48:48.207875  2233 linked_list-test-util.h:424] 36: -8069567904449560540
I20240502 01:48:48.208046  2233 linked_list-test-util.h:424] 37: 976240837010456613
I20240502 01:48:48.208204  2233 linked_list-test-util.h:424] 38: -8424694495239077850
I20240502 01:48:48.208364  2233 linked_list-test-util.h:424] 39: 5987352246314598439
I20240502 01:48:48.208519  2233 linked_list-test-util.h:424] 40: -3413723823423225816
I20240502 01:48:48.208690  2233 linked_list-test-util.h:424] 41: 5632084918036791337
I20240502 01:48:48.208851  2233 linked_list-test-util.h:424] 42: -3768850414212743126
I20240502 01:48:48.208997  2233 linked_list-test-util.h:424] 43: -7803266271391907797
I20240502 01:48:48.209141  2233 linked_list-test-util.h:424] 44: 1242542470068109356
I20240502 01:48:48.209283  2233 linked_list-test-util.h:424] 45: -8158533599669714899
I20240502 01:48:48.209426  2233 linked_list-test-util.h:424] 46: 887275141790302254
I20240502 01:48:48.209574  2233 linked_list-test-util.h:424] 47: -3147422190365573073
I20240502 01:48:48.209707  2233 linked_list-test-util.h:424] 48: 5898386551094444080
I20240502 01:48:48.209872  2233 linked_list-test-util.h:424] 49: -3502548781155090383
I20240502 01:48:48.328130  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614525112826 usec, L: 1 Expected Rows: 150
I20240502 01:48:48.348486  2233 linked_list-test-util.h:528] Calling callback on tserver 5ad6bd000714465092a095414dba4338
I20240502 01:48:48.393749  2233 meta_cache.cc:1510] marking tablet server 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777) as failed
W20240502 01:48:48.394124  2233 meta_cache.cc:302] tablet 3fe288ea889a4b3380330374e0568835: replica 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111)
I20240502 01:48:48.431339  2233 linked_list-test-util.h:767] Done collecting results (150 rows in 83.1287ms)
I20240502 01:48:48.432217  2233 linked_list-test-util.h:556] Snapshot: P: 1714614525112826 usec, L: 1 verified. Result: OK
I20240502 01:48:48.438802  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614526497252 usec, L: 1 Expected Rows: 1500
I20240502 01:48:48.489567  2233 linked_list-test-util.h:528] Calling callback on tserver 5ad6bd000714465092a095414dba4338
I20240502 01:48:48.656239  2233 linked_list-test-util.h:767] Done collecting results (1500 rows in 166.952ms)
I20240502 01:48:48.660179  2233 linked_list-test-util.h:556] Snapshot: P: 1714614526497252 usec, L: 1 verified. Result: OK
I20240502 01:48:48.667058  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614528182386 usec, L: 1 Expected Rows: 3100
I20240502 01:48:48.705880  2233 linked_list-test-util.h:528] Calling callback on tserver 5ad6bd000714465092a095414dba4338
I20240502 01:48:48.769838  2808 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/2886/stat: No such file or directory (error 2)
I20240502 01:48:48.887583  2233 linked_list-test-util.h:767] Done collecting results (3100 rows in 181.982ms)
I20240502 01:48:48.895422  2233 linked_list-test-util.h:556] Snapshot: P: 1714614528182386 usec, L: 1 verified. Result: OK
I20240502 01:48:48.902213  2233 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 3100
I20240502 01:48:48.993589  2807 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:48:49.152608  2358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:48:49.198568  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:48:49.456450  2233 scanner-internal.cc:458] Time spent opening tablet: real 0.554s	user 0.003s	sys 0.000s
I20240502 01:48:49.583305  2233 linked_list-test-util.h:767] Done collecting results (3100 rows in 126.705ms)
I20240502 01:48:49.587237  2233 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240502 01:48:49.588308  2233 linked_list-test-util.h:692] Successfully verified 3100 rows
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 69dc2bce073244fb8d08141f1c1a0bda, Rpc address: host: "127.2.46.66" port: 39777 (Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111))
I20240502 01:48:49.634397  2233 webui_checker.cc:88] shutting down CURL thread
W20240502 01:48:50.089680  2807 debug-util.cc:398] Leaking SignalData structure 0x7b08000d33e0 after lost signal to thread 2606
W20240502 01:48:50.194442  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:48:50.252285  2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 01:48:50.254915  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:48:51.762813  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:52.710737  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:52.836845  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:48:54.026682  2808 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4040 after lost signal to thread 2340
W20240502 01:48:54.027420  2808 debug-util.cc:398] Leaking SignalData structure 0x7b08000d47a0 after lost signal to thread 2467
W20240502 01:48:54.346639  2358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:48:54.425338  2358 consensus_peers.cc:485] T f6dc4786dff143caa9dee619976264e5 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:48:55.320326  2358 consensus_peers.cc:485] T 3fe288ea889a4b3380330374e0568835 P b16644ccdf9b430f88938474d8ab3060 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:48:55.361524  2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:48:55.366029  2624 consensus_peers.cc:485] T 8cdb154fb4b947e7aa46a952036daa40 P 5ad6bd000714465092a095414dba4338 -> Peer 69dc2bce073244fb8d08141f1c1a0bda (127.2.46.66:39777): Couldn't send request to peer 69dc2bce073244fb8d08141f1c1a0bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39777: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:48:56.299474  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2339
I20240502 01:48:56.346400  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2605
I20240502 01:48:56.386099  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2247
2024-05-02T01:48:56Z chronyd exiting
I20240502 01:48:56.434173  2233 test_util.cc:175] -----------------------------------------------
I20240502 01:48:56.434327  2233 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614514885052-2233-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (21415 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240502 01:48:56.437106  2233 test_util.cc:268] Using random seed: -832635560
I20240502 01:48:56.440573  2233 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 01:48:56.440696  2233 linked_list-test.cc:101] --------------
I20240502 01:48:56.440796  2233 linked_list-test.cc:102] 50 chains
I20240502 01:48:56.440897  2233 linked_list-test.cc:103] 3 tablets
I20240502 01:48:56.440986  2233 linked_list-test.cc:104] Mutations on
I20240502 01:48:56.441084  2233 linked_list-test.cc:105] --------------
I20240502 01:48:56.441293  2233 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:48:56.441404  2233 ts_itest-base.cc:113] --------------
I20240502 01:48:56.441498  2233 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:48:56.441586  2233 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:48:56.441689  2233 ts_itest-base.cc:116] --------------
2024-05-02T01:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:48:56Z Disabled control of system clock
I20240502 01:48:56.475663  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.46.126:39857
--webserver_interface=127.2.46.126
--webserver_port=0
--builtin_ntp_servers=127.2.46.84:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.46.126:39857
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:48:56.738586  2926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:56.739145  2926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:56.739626  2926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:56.765515  2926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:48:56.765796  2926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:56.766016  2926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:48:56.766224  2926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:48:56.797289  2926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.46.126:39857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.46.126:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.2.46.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:56.798348  2926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:56.800094  2926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:56.812964  2933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:56.813016  2932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:56.814394  2935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:57.793998  2926 server_base.cc:1034] running on GCE node
I20240502 01:48:57.796118  2926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:57.798858  2926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:57.800259  2926 hybrid_clock.cc:648] HybridClock initialized: now 1714614537800219 us; error 47 us; skew 500 ppm
I20240502 01:48:57.800961  2926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:57.811790  2926 webserver.cc:460] Webserver started at http://127.2.46.126:40251/ using document root <none> and password file <none>
I20240502 01:48:57.812640  2926 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:57.812835  2926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:57.813261  2926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:57.819303  2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/instance:
uuid: "8da42104790348c4b9f80b73990f6adc"
format_stamp: "Formatted at 2024-05-02 01:48:57 on dist-test-slave-hwlc"
I20240502 01:48:57.820372  2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal/instance:
uuid: "8da42104790348c4b9f80b73990f6adc"
format_stamp: "Formatted at 2024-05-02 01:48:57 on dist-test-slave-hwlc"
I20240502 01:48:57.826468  2926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.001s	sys 0.005s
I20240502 01:48:57.831333  2942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:57.832285  2926 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:48:57.832556  2926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
uuid: "8da42104790348c4b9f80b73990f6adc"
format_stamp: "Formatted at 2024-05-02 01:48:57 on dist-test-slave-hwlc"
I20240502 01:48:57.832834  2926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:57.885309  2926 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:57.886562  2926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:57.886945  2926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:57.947319  2926 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.126:39857
I20240502 01:48:57.947394  2993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.126:39857 every 8 connection(s)
I20240502 01:48:57.949798  2926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/data/info.pb
I20240502 01:48:57.951799  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 2926
I20240502 01:48:57.952325  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/master-0/wal/instance
I20240502 01:48:57.954890  2994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:57.976459  2994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc: Bootstrap starting.
I20240502 01:48:57.981079  2994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:57.982455  2994 log.cc:824] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:57.986150  2994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc: No bootstrap required, opened a new log
I20240502 01:48:58.003760  2994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } }
I20240502 01:48:58.004323  2994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:58.004518  2994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da42104790348c4b9f80b73990f6adc, State: Initialized, Role: FOLLOWER
I20240502 01:48:58.005118  2994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } }
I20240502 01:48:58.005558  2994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:58.005784  2994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:58.006040  2994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:58.009678  2994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } }
I20240502 01:48:58.010248  2994 leader_election.cc:302] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da42104790348c4b9f80b73990f6adc; no voters: 
I20240502 01:48:58.011668  2994 leader_election.cc:288] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:48:58.012392  2999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:58.014957  2999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [term 1 LEADER]: Becoming Leader. State: Replica: 8da42104790348c4b9f80b73990f6adc, State: Running, Role: LEADER
I20240502 01:48:58.015580  2999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } }
I20240502 01:48:58.016013  2994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:58.025632  3000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da42104790348c4b9f80b73990f6adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } } }
I20240502 01:48:58.026278  3000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:58.027567  3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da42104790348c4b9f80b73990f6adc. Latest consensus state: current_term: 1 leader_uuid: "8da42104790348c4b9f80b73990f6adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da42104790348c4b9f80b73990f6adc" member_type: VOTER last_known_addr { host: "127.2.46.126" port: 39857 } } }
I20240502 01:48:58.028281  3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:58.030210  3009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:58.040419  3009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:58.053933  3009 catalog_manager.cc:1347] Generated new cluster ID: 27cd8c8b17ed4433a539c7970d4bb38e
I20240502 01:48:58.054132  3009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:58.081626  3009 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:58.082796  3009 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:58.098045  3009 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8da42104790348c4b9f80b73990f6adc: Generated new TSK 0
I20240502 01:48:58.098785  3009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:58.106173  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.65:0
--local_ip_for_outbound_sockets=127.2.46.65
--webserver_interface=127.2.46.65
--webserver_port=0
--tserver_master_addrs=127.2.46.126:39857
--builtin_ntp_servers=127.2.46.84:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 01:48:58.387668  3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:48:58.388134  3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:48:58.388651  3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:48:58.414641  3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:48:58.415395  3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.65
I20240502 01:48:58.444154  3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.2.46.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:39857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.65
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:48:58.445128  3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:58.446440  3018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:58.462035  3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:58.464519  3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:59.710062  3026 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1249 milliseconds
I20240502 01:48:59.710111  3018 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20240502 01:48:58.462812  3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:59.714680  3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:48:59.717381  3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:48:59.718824  3018 hybrid_clock.cc:648] HybridClock initialized: now 1714614539718761 us; error 73 us; skew 500 ppm
I20240502 01:48:59.719604  3018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:59.725890  3018 webserver.cc:460] Webserver started at http://127.2.46.65:41477/ using document root <none> and password file <none>
I20240502 01:48:59.726681  3018 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:59.726876  3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:59.727300  3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:59.731962  3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/instance:
uuid: "761697cb092a4991a2407bbf0558d488"
format_stamp: "Formatted at 2024-05-02 01:48:59 on dist-test-slave-hwlc"
I20240502 01:48:59.732930  3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal/instance:
uuid: "761697cb092a4991a2407bbf0558d488"
format_stamp: "Formatted at 2024-05-02 01:48:59 on dist-test-slave-hwlc"
I20240502 01:48:59.739737  3018 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 01:48:59.744657  3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:59.745654  3018 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
I20240502 01:48:59.745919  3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
uuid: "761697cb092a4991a2407bbf0558d488"
format_stamp: "Formatted at 2024-05-02 01:48:59 on dist-test-slave-hwlc"
I20240502 01:48:59.746201  3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:59.788705  3018 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:48:59.790050  3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:59.790422  3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:59.792963  3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:59.796499  3018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:59.796679  3018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:59.796880  3018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:59.797011  3018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:48:59.943058  3018 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.65:43755
I20240502 01:48:59.943233  3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.65:43755 every 8 connection(s)
I20240502 01:48:59.945510  3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:48:59.954277  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 3018
I20240502 01:48:59.954756  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal/instance
I20240502 01:48:59.962211  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.66:0
--local_ip_for_outbound_sockets=127.2.46.66
--webserver_interface=127.2.46.66
--webserver_port=0
--tserver_master_addrs=127.2.46.126:39857
--builtin_ntp_servers=127.2.46.84:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 01:48:59.971602  3147 heartbeater.cc:347] Connected to a master server at 127.2.46.126:39857
I20240502 01:48:59.972033  3147 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:59.973089  3147 heartbeater.cc:511] Master 127.2.46.126:39857 requested a full tablet report, sending...
I20240502 01:48:59.975752  2959 ts_manager.cc:194] Registered new tserver with Master: 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755)
I20240502 01:48:59.978060  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.65:48063
W20240502 01:49:00.250226  3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:49:00.250663  3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:49:00.251137  3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:49:00.278021  3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:49:00.278877  3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.66
I20240502 01:49:00.307669  3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.2.46.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:39857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.66
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:49:00.308723  3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:49:00.310205  3151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:00.324083  3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:00.981699  3147 heartbeater.cc:503] Master 127.2.46.126:39857 was elected leader, sending a full tablet report...
W20240502 01:49:00.324112  3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.324090  3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:00.326539  3151 server_base.cc:1034] running on GCE node
I20240502 01:49:01.286002  3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:49:01.288614  3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:49:01.290119  3151 hybrid_clock.cc:648] HybridClock initialized: now 1714614541290076 us; error 52 us; skew 500 ppm
I20240502 01:49:01.290863  3151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:01.297259  3151 webserver.cc:460] Webserver started at http://127.2.46.66:42497/ using document root <none> and password file <none>
I20240502 01:49:01.298063  3151 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:01.298230  3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:01.298627  3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:01.303010  3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/instance:
uuid: "672e6eb474ad47f7886b29cce038cb60"
format_stamp: "Formatted at 2024-05-02 01:49:01 on dist-test-slave-hwlc"
I20240502 01:49:01.304054  3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal/instance:
uuid: "672e6eb474ad47f7886b29cce038cb60"
format_stamp: "Formatted at 2024-05-02 01:49:01 on dist-test-slave-hwlc"
I20240502 01:49:01.310333  3151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:49:01.315258  3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:01.316159  3151 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:49:01.316423  3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
uuid: "672e6eb474ad47f7886b29cce038cb60"
format_stamp: "Formatted at 2024-05-02 01:49:01 on dist-test-slave-hwlc"
I20240502 01:49:01.316695  3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:01.374114  3151 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:49:01.375407  3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:49:01.375836  3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:01.378127  3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:01.381978  3151 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:01.382166  3151 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:01.382349  3151 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:01.382477  3151 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:01.507959  3151 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.66:39819
I20240502 01:49:01.508040  3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.66:39819 every 8 connection(s)
I20240502 01:49:01.510442  3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/data/info.pb
I20240502 01:49:01.513324  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 3151
I20240502 01:49:01.513760  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-1/wal/instance
I20240502 01:49:01.520725  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.67:0
--local_ip_for_outbound_sockets=127.2.46.67
--webserver_interface=127.2.46.67
--webserver_port=0
--tserver_master_addrs=127.2.46.126:39857
--builtin_ntp_servers=127.2.46.84:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 01:49:01.533524  3280 heartbeater.cc:347] Connected to a master server at 127.2.46.126:39857
I20240502 01:49:01.533972  3280 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:01.535197  3280 heartbeater.cc:511] Master 127.2.46.126:39857 requested a full tablet report, sending...
I20240502 01:49:01.537460  2959 ts_manager.cc:194] Registered new tserver with Master: 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819)
I20240502 01:49:01.538755  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.66:38975
W20240502 01:49:01.796829  3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:49:01.797250  3284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:49:01.797730  3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:49:01.823685  3284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:49:01.824460  3284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.67
I20240502 01:49:01.854446  3284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.2.46.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:39857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.67
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:49:01.855422  3284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:49:01.856827  3284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:01.873000  3293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:02.542017  3280 heartbeater.cc:503] Master 127.2.46.126:39857 was elected leader, sending a full tablet report...
W20240502 01:49:01.873025  3290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:01.873272  3291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:01.875025  3284 server_base.cc:1034] running on GCE node
I20240502 01:49:02.901101  3284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:49:02.903242  3284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:49:02.904549  3284 hybrid_clock.cc:648] HybridClock initialized: now 1714614542904507 us; error 53 us; skew 500 ppm
I20240502 01:49:02.905383  3284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:02.915199  3284 webserver.cc:460] Webserver started at http://127.2.46.67:35945/ using document root <none> and password file <none>
I20240502 01:49:02.916137  3284 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:02.916326  3284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:02.916740  3284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:02.923300  3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/instance:
uuid: "f5f99849444343c7aad4e19e981d4e9e"
format_stamp: "Formatted at 2024-05-02 01:49:02 on dist-test-slave-hwlc"
I20240502 01:49:02.924362  3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal/instance:
uuid: "f5f99849444343c7aad4e19e981d4e9e"
format_stamp: "Formatted at 2024-05-02 01:49:02 on dist-test-slave-hwlc"
I20240502 01:49:02.930859  3284 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:49:02.935679  3300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:02.936617  3284 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:49:02.936893  3284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
uuid: "f5f99849444343c7aad4e19e981d4e9e"
format_stamp: "Formatted at 2024-05-02 01:49:02 on dist-test-slave-hwlc"
I20240502 01:49:02.937175  3284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:02.986899  3284 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:49:02.988255  3284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:49:02.988631  3284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:02.991185  3284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:02.995093  3284 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:02.995255  3284 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:02.995474  3284 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:02.995607  3284 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:03.117730  3284 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.67:37711
I20240502 01:49:03.117827  3412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.67:37711 every 8 connection(s)
I20240502 01:49:03.120115  3284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/data/info.pb
I20240502 01:49:03.124230  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 3284
I20240502 01:49:03.124729  2233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-2/wal/instance
I20240502 01:49:03.139601  3413 heartbeater.cc:347] Connected to a master server at 127.2.46.126:39857
I20240502 01:49:03.139973  3413 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:03.140784  3413 heartbeater.cc:511] Master 127.2.46.126:39857 requested a full tablet report, sending...
I20240502 01:49:03.142710  2958 ts_manager.cc:194] Registered new tserver with Master: f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711)
I20240502 01:49:03.144052  2958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.67:40703
I20240502 01:49:03.144409  2233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:49:03.174777  2958 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54804:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240502 01:49:03.193751  2958 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:49:03.245007  3215 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd80e0f3d4f486b80092fafd7a421fc (DEFAULT_TABLE table=TestTable [id=02e67ebdc6184dd5a93bd6a3fd73cdde]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:49:03.245725  3082 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd80e0f3d4f486b80092fafd7a421fc (DEFAULT_TABLE table=TestTable [id=02e67ebdc6184dd5a93bd6a3fd73cdde]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:49:03.246621  3215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd80e0f3d4f486b80092fafd7a421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:03.246721  3348 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd80e0f3d4f486b80092fafd7a421fc (DEFAULT_TABLE table=TestTable [id=02e67ebdc6184dd5a93bd6a3fd73cdde]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:49:03.248127  3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd80e0f3d4f486b80092fafd7a421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:03.248579  3348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd80e0f3d4f486b80092fafd7a421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:03.268631  3432 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Bootstrap starting.
I20240502 01:49:03.274030  3432 tablet_bootstrap.cc:654] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:03.275624  3433 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Bootstrap starting.
I20240502 01:49:03.278831  3434 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Bootstrap starting.
I20240502 01:49:03.279352  3432 log.cc:824] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Log is configured to *not* fsync() on all Append() calls
I20240502 01:49:03.281855  3433 tablet_bootstrap.cc:654] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:03.283558  3433 log.cc:824] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Log is configured to *not* fsync() on all Append() calls
I20240502 01:49:03.284682  3432 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: No bootstrap required, opened a new log
I20240502 01:49:03.285224  3432 ts_tablet_manager.cc:1367] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.003s
I20240502 01:49:03.285413  3434 tablet_bootstrap.cc:654] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:03.287509  3434 log.cc:824] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:49:03.287806  3433 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: No bootstrap required, opened a new log
I20240502 01:49:03.288157  3433 ts_tablet_manager.cc:1367] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.004s
I20240502 01:49:03.292831  3434 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: No bootstrap required, opened a new log
I20240502 01:49:03.293359  3434 ts_tablet_manager.cc:1367] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Time spent bootstrapping tablet: real 0.015s	user 0.010s	sys 0.004s
I20240502 01:49:03.307305  3433 raft_consensus.cc:348] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.307889  3433 raft_consensus.cc:374] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:03.308174  3433 raft_consensus.cc:729] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761697cb092a4991a2407bbf0558d488, State: Initialized, Role: FOLLOWER
I20240502 01:49:03.309155  3433 consensus_queue.cc:260] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.313102  3433 ts_tablet_manager.cc:1398] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Time spent starting tablet: real 0.025s	user 0.016s	sys 0.008s
I20240502 01:49:03.313385  3432 raft_consensus.cc:348] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.314172  3432 raft_consensus.cc:374] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:03.314421  3432 raft_consensus.cc:729] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672e6eb474ad47f7886b29cce038cb60, State: Initialized, Role: FOLLOWER
I20240502 01:49:03.315122  3432 consensus_queue.cc:260] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.319463  3434 raft_consensus.cc:348] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.320251  3434 raft_consensus.cc:374] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:03.320561  3434 raft_consensus.cc:729] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f99849444343c7aad4e19e981d4e9e, State: Initialized, Role: FOLLOWER
I20240502 01:49:03.321489  3434 consensus_queue.cc:260] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.321987  3432 ts_tablet_manager.cc:1398] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Time spent starting tablet: real 0.036s	user 0.024s	sys 0.011s
I20240502 01:49:03.324345  3413 heartbeater.cc:503] Master 127.2.46.126:39857 was elected leader, sending a full tablet report...
I20240502 01:49:03.325366  3434 ts_tablet_manager.cc:1398] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Time spent starting tablet: real 0.032s	user 0.025s	sys 0.005s
W20240502 01:49:03.374307  3414 tablet.cc:2369] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:49:03.453382  3148 tablet.cc:2369] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:49:03.516238  3281 tablet.cc:2369] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:49:03.634263  3440 raft_consensus.cc:482] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:03.634668  3440 raft_consensus.cc:504] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.636708  3440 leader_election.cc:288] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755), f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711)
I20240502 01:49:03.647897  3102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd80e0f3d4f486b80092fafd7a421fc" candidate_uuid: "672e6eb474ad47f7886b29cce038cb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761697cb092a4991a2407bbf0558d488" is_pre_election: true
I20240502 01:49:03.648264  3368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd80e0f3d4f486b80092fafd7a421fc" candidate_uuid: "672e6eb474ad47f7886b29cce038cb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f99849444343c7aad4e19e981d4e9e" is_pre_election: true
I20240502 01:49:03.648730  3102 raft_consensus.cc:2454] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672e6eb474ad47f7886b29cce038cb60 in term 0.
I20240502 01:49:03.648994  3368 raft_consensus.cc:2454] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672e6eb474ad47f7886b29cce038cb60 in term 0.
I20240502 01:49:03.649942  3169 leader_election.cc:302] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672e6eb474ad47f7886b29cce038cb60, 761697cb092a4991a2407bbf0558d488; no voters: 
I20240502 01:49:03.650553  3440 raft_consensus.cc:2781] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:03.650856  3440 raft_consensus.cc:482] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:03.651105  3440 raft_consensus.cc:3037] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:03.654773  3440 raft_consensus.cc:504] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.655967  3440 leader_election.cc:288] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [CANDIDATE]: Term 1 election: Requested vote from peers 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755), f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711)
I20240502 01:49:03.656504  3102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd80e0f3d4f486b80092fafd7a421fc" candidate_uuid: "672e6eb474ad47f7886b29cce038cb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761697cb092a4991a2407bbf0558d488"
I20240502 01:49:03.656705  3368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd80e0f3d4f486b80092fafd7a421fc" candidate_uuid: "672e6eb474ad47f7886b29cce038cb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f99849444343c7aad4e19e981d4e9e"
I20240502 01:49:03.656847  3102 raft_consensus.cc:3037] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:03.657070  3368 raft_consensus.cc:3037] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:03.660641  3102 raft_consensus.cc:2454] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672e6eb474ad47f7886b29cce038cb60 in term 1.
I20240502 01:49:03.661101  3368 raft_consensus.cc:2454] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672e6eb474ad47f7886b29cce038cb60 in term 1.
I20240502 01:49:03.661273  3169 leader_election.cc:302] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672e6eb474ad47f7886b29cce038cb60, 761697cb092a4991a2407bbf0558d488; no voters: 
I20240502 01:49:03.661847  3440 raft_consensus.cc:2781] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:03.663218  3440 raft_consensus.cc:686] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [term 1 LEADER]: Becoming Leader. State: Replica: 672e6eb474ad47f7886b29cce038cb60, State: Running, Role: LEADER
I20240502 01:49:03.664017  3440 consensus_queue.cc:237] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:03.673921  2957 catalog_manager.cc:5484] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66). New cstate: current_term: 1 leader_uuid: "672e6eb474ad47f7886b29cce038cb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:03.690150  2233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:49:03.693054  2233 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 761697cb092a4991a2407bbf0558d488 to finish bootstrapping
I20240502 01:49:03.703181  2233 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 672e6eb474ad47f7886b29cce038cb60 to finish bootstrapping
I20240502 01:49:03.712795  2233 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f5f99849444343c7aad4e19e981d4e9e to finish bootstrapping
I20240502 01:49:03.721416  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 3018
I20240502 01:49:03.748692  2233 linked_list-test-util.h:401] Running for: 5.000s
I20240502 01:49:03.749013  2233 linked_list-test-util.h:402] Sampling every 1666666 us
I20240502 01:49:03.749186  2233 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240502 01:49:03.786763  3368 raft_consensus.cc:1261] T 6cd80e0f3d4f486b80092fafd7a421fc P f5f99849444343c7aad4e19e981d4e9e [term 1 FOLLOWER]: Refusing update from remote peer 672e6eb474ad47f7886b29cce038cb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:49:03.786801  3169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.65:43755: connect: Connection refused (error 111)
I20240502 01:49:03.788693  3444 consensus_queue.cc:1035] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:03.790001  3169 consensus_peers.cc:485] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 -> Peer 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755): Couldn't send request to peer 761697cb092a4991a2407bbf0558d488. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.65:43755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:05.426492  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061178025447424 Inserted count: 2900
W20240502 01:49:05.838331  3169 consensus_peers.cc:485] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 -> Peer 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755): Couldn't send request to peer 761697cb092a4991a2407bbf0558d488. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.65:43755: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:49:07.100625  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061184829599744 Inserted count: 5900
W20240502 01:49:08.212358  3169 consensus_peers.cc:485] T 6cd80e0f3d4f486b80092fafd7a421fc P 672e6eb474ad47f7886b29cce038cb60 -> Peer 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755): Couldn't send request to peer 761697cb092a4991a2407bbf0558d488. Status: Network error: Client connection negotiation failed: client connection to 127.2.46.65:43755: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:49:08.756126  2233 linked_list-test-util.h:417] Sample at HT timestamp: 7023061191652085760 Inserted count: 8950
I20240502 01:49:08.756311  2233 linked_list-test-util.h:421] Finished inserting list. Added 8950 in chain
I20240502 01:49:08.756455  2233 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 01:49:08.756577  2233 linked_list-test-util.h:424] 0: -3977474507878629376
I20240502 01:49:08.756691  2233 linked_list-test-util.h:424] 1: 5340499778046918657
I20240502 01:49:08.756810  2233 linked_list-test-util.h:424] 2: -3788551484713795582
I20240502 01:49:08.756937  2233 linked_list-test-util.h:424] 3: 5529422801211752451
I20240502 01:49:08.757055  2233 linked_list-test-util.h:424] 4: -7576962231939170300
I20240502 01:49:08.757177  2233 linked_list-test-util.h:424] 5: 1740730579009667077
I20240502 01:49:08.757297  2233 linked_list-test-util.h:424] 6: -7388039208774336506
I20240502 01:49:08.757413  2233 linked_list-test-util.h:424] 7: 1929512864686211079
I20240502 01:49:08.757532  2233 linked_list-test-util.h:424] 8: 7270294117709840392
I20240502 01:49:08.757648  2233 linked_list-test-util.h:424] 9: -1858757145050873847
I20240502 01:49:08.757766  2233 linked_list-test-util.h:424] 10: 7459217140874674186
I20240502 01:49:08.757890  2233 linked_list-test-util.h:424] 11: -1669974859374329845
I20240502 01:49:08.758008  2233 linked_list-test-util.h:424] 12: 3670524918672588812
I20240502 01:49:08.758126  2233 linked_list-test-util.h:424] 13: -5458244869111414771
I20240502 01:49:08.758246  2233 linked_list-test-util.h:424] 14: 3859307204349132814
I20240502 01:49:08.758365  2233 linked_list-test-util.h:424] 15: -5269462583434870769
I20240502 01:49:08.758484  2233 linked_list-test-util.h:424] 16: 71037194612047888
I20240502 01:49:08.758601  2233 linked_list-test-util.h:424] 17: -9058014068148666351
I20240502 01:49:08.758716  2233 linked_list-test-util.h:424] 18: 259819480288591890
I20240502 01:49:08.758832  2233 linked_list-test-util.h:424] 19: -8869231782472122349
I20240502 01:49:08.758952  2233 linked_list-test-util.h:424] 20: -3528450529448493036
I20240502 01:49:08.759091  2233 linked_list-test-util.h:424] 21: 5789101544012054549
I20240502 01:49:08.759238  2233 linked_list-test-util.h:424] 22: -3339668243771949034
I20240502 01:49:08.759373  2233 linked_list-test-util.h:424] 23: 5978024567176888343
I20240502 01:49:08.759510  2233 linked_list-test-util.h:424] 24: -7128219728485744616
I20240502 01:49:08.759644  2233 linked_list-test-util.h:424] 25: 2189613819951513625
I20240502 01:49:08.759804  2233 linked_list-test-util.h:424] 26: -6939437442809200614
I20240502 01:49:08.759941  2233 linked_list-test-util.h:424] 27: -1598656189785571301
I20240502 01:49:08.760074  2233 linked_list-test-util.h:424] 28: 7718895883674976284
I20240502 01:49:08.760206  2233 linked_list-test-util.h:424] 29: -1409873904109027299
I20240502 01:49:08.760334  2233 linked_list-test-util.h:424] 30: 7907818906839810078
I20240502 01:49:08.760469  2233 linked_list-test-util.h:424] 31: -5198425388822822881
I20240502 01:49:08.760581  2233 linked_list-test-util.h:424] 32: 4119408159614435360
I20240502 01:49:08.760696  2233 linked_list-test-util.h:424] 33: -5009643103146278879
I20240502 01:49:08.760814  2233 linked_list-test-util.h:424] 34: 4308331182779269154
I20240502 01:49:08.760934  2233 linked_list-test-util.h:424] 35: -8798053850371653597
I20240502 01:49:08.761049  2233 linked_list-test-util.h:424] 36: 519638960577183780
I20240502 01:49:08.761168  2233 linked_list-test-util.h:424] 37: -8609130827206819803
I20240502 01:49:08.761281  2233 linked_list-test-util.h:424] 38: 708561983742017574
I20240502 01:49:08.761397  2233 linked_list-test-util.h:424] 39: 6049202499277357095
I20240502 01:49:08.761512  2233 linked_list-test-util.h:424] 40: -3079848763483357144
I20240502 01:49:08.761626  2233 linked_list-test-util.h:424] 41: 6238125522442190889
I20240502 01:49:08.761742  2233 linked_list-test-util.h:424] 42: -2890925740318523350
I20240502 01:49:08.761857  2233 linked_list-test-util.h:424] 43: 2449433300240105515
I20240502 01:49:08.761977  2233 linked_list-test-util.h:424] 44: -6679336487543898068
I20240502 01:49:08.762094  2233 linked_list-test-util.h:424] 45: 2638356323404939309
I20240502 01:49:08.762208  2233 linked_list-test-util.h:424] 46: -6490413464379064274
I20240502 01:49:08.762339  2233 linked_list-test-util.h:424] 47: -1150054423820435409
I20240502 01:49:08.762460  2233 linked_list-test-util.h:424] 48: 8167919862105112624
I20240502 01:49:08.762579  2233 linked_list-test-util.h:424] 49: -961131400655601615
I20240502 01:49:08.910753  2233 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
/tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.46.65:43755
--local_ip_for_outbound_sockets=127.2.46.65
--tserver_master_addrs=127.2.46.126:39857
--webserver_port=41477
--webserver_interface=127.2.46.65
--builtin_ntp_servers=127.2.46.84:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 01:49:09.208360  3481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:49:09.208781  3481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:49:09.209336  3481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:49:09.236972  3481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:49:09.237805  3481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.46.65
W20240502 01:49:09.263845  3169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.65:43755: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 01:49:09.277133  3481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.46.84:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.46.65:43755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.2.46.65
--webserver_port=41477
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.2.46.126:39857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.46.65
--log_dir=/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on 1a15035a6830
build id 84
TSAN enabled
I20240502 01:49:09.278179  3481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:49:09.279667  3481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:09.294729  3490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:09.294848  3489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:09.295929  3492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:10.392380  3491 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1096 milliseconds
I20240502 01:49:10.392513  3481 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 01:49:10.393613  3481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:49:10.395602  3481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:49:10.396981  3481 hybrid_clock.cc:648] HybridClock initialized: now 1714614550396940 us; error 50 us; skew 500 ppm
I20240502 01:49:10.397730  3481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:10.408490  3481 webserver.cc:460] Webserver started at http://127.2.46.65:41477/ using document root <none> and password file <none>
I20240502 01:49:10.409368  3481 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:10.409605  3481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:10.416572  3481 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:49:10.420765  3502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:49:10.421785  3481 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 01:49:10.422114  3481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
uuid: "761697cb092a4991a2407bbf0558d488"
format_stamp: "Formatted at 2024-05-02 01:48:59 on dist-test-slave-hwlc"
I20240502 01:49:10.424388  3481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:10.481789  3481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:49:10.483271  3481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:49:10.483738  3481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:10.486064  3481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:10.491375  3509 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:49:10.498529  3481 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:49:10.498740  3481 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.001s
I20240502 01:49:10.498980  3481 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:49:10.503399  3481 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:49:10.503579  3481 ts_tablet_manager.cc:569] Time spent register tablets: real 0.005s	user 0.001s	sys 0.000s
I20240502 01:49:10.504010  3509 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Bootstrap starting.
I20240502 01:49:10.549773  3509 log.cc:824] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Log is configured to *not* fsync() on all Append() calls
I20240502 01:49:10.555368  3509 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:49:10.555780  3509 log.cc:1225] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/wal/wals/6cd80e0f3d4f486b80092fafd7a421fc.recovery-1714614550555675
I20240502 01:49:10.555964  3509 tablet_bootstrap.cc:492] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Bootstrap complete.
I20240502 01:49:10.556423  3509 ts_tablet_manager.cc:1367] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Time spent bootstrapping tablet: real 0.053s	user 0.028s	sys 0.022s
I20240502 01:49:10.582248  3509 raft_consensus.cc:348] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:10.584765  3509 raft_consensus.cc:729] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761697cb092a4991a2407bbf0558d488, State: Initialized, Role: FOLLOWER
I20240502 01:49:10.585821  3509 consensus_queue.cc:260] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:10.590255  3509 ts_tablet_manager.cc:1398] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Time spent starting tablet: real 0.034s	user 0.024s	sys 0.008s
I20240502 01:49:10.649960  3481 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.65:43755
I20240502 01:49:10.650152  3617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.46.65:43755 every 8 connection(s)
I20240502 01:49:10.652308  3481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBUgo_/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614514885052-2233-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:49:10.655514  2233 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu as pid 3481
W20240502 01:49:10.656132  3619 tablet.cc:2369] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:49:10.671773  3618 heartbeater.cc:347] Connected to a master server at 127.2.46.126:39857
I20240502 01:49:10.672266  3618 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:10.673350  3618 heartbeater.cc:511] Master 127.2.46.126:39857 requested a full tablet report, sending...
I20240502 01:49:10.677309  2956 ts_manager.cc:194] Re-registered known tserver with Master: 761697cb092a4991a2407bbf0558d488 (127.2.46.65:43755)
I20240502 01:49:10.680634  2956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.46.65:44145
I20240502 01:49:10.694242  2233 cluster_itest_util.cc:256] Not converged past 179 yet: 0.0 1.185 1.185
I20240502 01:49:10.799702  2233 cluster_itest_util.cc:256] Not converged past 179 yet: 0.0 1.185 1.185
I20240502 01:49:10.842788  3618 heartbeater.cc:503] Master 127.2.46.126:39857 was elected leader, sending a full tablet report...
I20240502 01:49:11.070159  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 3151
W20240502 01:49:11.079560  3628 log.cc:924] Time spent T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488: Append to log took a long time: real 0.146s	user 0.123s	sys 0.012s
W20240502 01:49:11.146185  3505 connection.cc:535] server connection from 127.2.46.66:46541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:49:11.172005  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 3284
I20240502 01:49:11.237757  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614545416369 usec, L: 1 Expected Rows: 2900
I20240502 01:49:11.249691  2233 meta_cache.cc:1510] marking tablet server 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) as failed
W20240502 01:49:11.250101  2233 meta_cache.cc:302] tablet 6cd80e0f3d4f486b80092fafd7a421fc: replica 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.2.46.66:39819: connect: Connection refused (error 111)
I20240502 01:49:11.258172  2233 meta_cache.cc:1510] marking tablet server f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711) as failed
I20240502 01:49:11.423997  2233 linked_list-test-util.h:528] Calling callback on tserver 761697cb092a4991a2407bbf0558d488
I20240502 01:49:11.862555  2233 linked_list-test-util.h:767] Done collecting results (2900 rows in 438.852ms)
I20240502 01:49:11.871356  2233 linked_list-test-util.h:556] Snapshot: P: 1714614545416369 usec, L: 1 verified. Result: OK
I20240502 01:49:11.880120  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614547077539 usec, L: 1 Expected Rows: 5900
I20240502 01:49:11.905751  2233 meta_cache.cc:1510] marking tablet server 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) as failed
I20240502 01:49:11.911465  2233 meta_cache.cc:1510] marking tablet server f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711) as failed
I20240502 01:49:11.970422  2233 linked_list-test-util.h:528] Calling callback on tserver 761697cb092a4991a2407bbf0558d488
I20240502 01:49:12.340112  3627 mvcc.cc:205] Tried to move back new op lower bound from 7023061191962206208 to 7023061170853515264. Current Snapshot: MvccSnapshot[applied={T|T < 7023061191962206208}]
I20240502 01:49:12.350713  3637 raft_consensus.cc:482] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 672e6eb474ad47f7886b29cce038cb60)
I20240502 01:49:12.351457  3637 raft_consensus.cc:504] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761697cb092a4991a2407bbf0558d488" member_type: VOTER last_known_addr { host: "127.2.46.65" port: 43755 } } peers { permanent_uuid: "672e6eb474ad47f7886b29cce038cb60" member_type: VOTER last_known_addr { host: "127.2.46.66" port: 39819 } } peers { permanent_uuid: "f5f99849444343c7aad4e19e981d4e9e" member_type: VOTER last_known_addr { host: "127.2.46.67" port: 37711 } }
I20240502 01:49:12.370261  3637 leader_election.cc:288] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819), f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711)
W20240502 01:49:12.393524  3505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.46.66:39819: connect: Connection refused (error 111)
W20240502 01:49:12.406883  3504 leader_election.cc:334] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711): Network error: Client connection negotiation failed: client connection to 127.2.46.67:37711: connect: Connection refused (error 111)
W20240502 01:49:12.409008  3505 leader_election.cc:334] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819): Network error: Client connection negotiation failed: client connection to 127.2.46.66:39819: connect: Connection refused (error 111)
I20240502 01:49:12.409394  3505 leader_election.cc:302] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 761697cb092a4991a2407bbf0558d488; no voters: 672e6eb474ad47f7886b29cce038cb60, f5f99849444343c7aad4e19e981d4e9e
I20240502 01:49:12.410226  3637 raft_consensus.cc:2726] T 6cd80e0f3d4f486b80092fafd7a421fc P 761697cb092a4991a2407bbf0558d488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:12.482906  2233 linked_list-test-util.h:767] Done collecting results (5900 rows in 512.696ms)
I20240502 01:49:12.490361  2233 linked_list-test-util.h:556] Snapshot: P: 1714614547077539 usec, L: 1 verified. Result: OK
I20240502 01:49:12.496829  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614548743185 usec, L: 1 Expected Rows: 8950
I20240502 01:49:12.499840  2233 meta_cache.cc:1510] marking tablet server 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) as failed
W20240502 01:49:12.500147  2233 meta_cache.cc:302] tablet 6cd80e0f3d4f486b80092fafd7a421fc: replica 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.2.46.66:39819: connect: Connection refused (error 111)
I20240502 01:49:12.505323  2233 meta_cache.cc:1510] marking tablet server f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711) as failed
I20240502 01:49:12.541759  2233 linked_list-test-util.h:528] Calling callback on tserver 761697cb092a4991a2407bbf0558d488
I20240502 01:49:12.948623  2233 linked_list-test-util.h:767] Done collecting results (8950 rows in 407.071ms)
I20240502 01:49:12.968750  2233 linked_list-test-util.h:556] Snapshot: P: 1714614548743185 usec, L: 1 verified. Result: OK
I20240502 01:49:12.976902  2233 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 8950
I20240502 01:49:12.979234  2233 meta_cache.cc:1510] marking tablet server 672e6eb474ad47f7886b29cce038cb60 (127.2.46.66:39819) as failed
I20240502 01:49:12.983866  2233 meta_cache.cc:1510] marking tablet server f5f99849444343c7aad4e19e981d4e9e (127.2.46.67:37711) as failed
I20240502 01:49:12.987907  2233 linked_list-test-util.h:528] Calling callback on tserver 761697cb092a4991a2407bbf0558d488
I20240502 01:49:13.452096  2233 linked_list-test-util.h:767] Done collecting results (8950 rows in 464.425ms)
I20240502 01:49:13.473274  2233 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240502 01:49:13.476150  2233 linked_list-test-util.h:692] Successfully verified 8950 rows
I20240502 01:49:13.478324  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 3481
I20240502 01:49:13.520367  2233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJBUgo_/build/tsan/bin/kudu with pid 2926
2024-05-02T01:49:13Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (17136 ms)
[----------] 2 tests from LinkedListTest (38552 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (38552 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST
I20240502 01:49:13.574172  2233 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 3 messages since previous log ~1 seconds ago