LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 57c0b8293f214a6ba5bec482f225d3b3, Rpc address: host: "127.1.164.195" port: 33183 (Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111))
I20240426 09:58:19.431623  1683 webui_checker.cc:88] shutting down CURL thread
W20240426 09:58:19.939992  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 09:58:20.555303  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240426 09:58:20.720772  2433 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 429 ops behind the committed index  [suppressed 5 similar messages]
I20240426 09:58:20.982959  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1785
I20240426 09:58:20.984316  1907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.266331  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1785
W20240426 09:58:21.326135  1710 connection.cc:535] server connection from 127.1.164.193:45623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240426 09:58:21.326524  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1916
W20240426 09:58:21.326839  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345): Couldn't send request to peer 56e7c87ce920461cba4d6cccc793ebd4. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:38345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:58:21.326957  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345): Couldn't send request to peer 56e7c87ce920461cba4d6cccc793ebd4. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:38345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:58:21.327505  2038 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.579411  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1916
I20240426 09:58:21.625548  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1695
I20240426 09:58:21.626680  1756 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.748534  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1695
2024-04-26T09:58:21Z chronyd exiting
I20240426 09:58:21.789584  1683 test_util.cc:175] -----------------------------------------------
I20240426 09:58:21.789702  1683 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 09:57:41.861470  1683 test_util.cc:268] Using random seed: -280939471
I20240426 09:57:41.873297  1683 linked_list-test.cc:100] Linked List Test Configuration:
I20240426 09:57:41.873384  1683 linked_list-test.cc:101] --------------
I20240426 09:57:41.873435  1683 linked_list-test.cc:102] 50 chains
I20240426 09:57:41.873505  1683 linked_list-test.cc:103] 3 tablets
I20240426 09:57:41.873571  1683 linked_list-test.cc:104] Mutations on
I20240426 09:57:41.873628  1683 linked_list-test.cc:105] --------------
I20240426 09:57:41.873879  1683 ts_itest-base.cc:112] Starting cluster with:
I20240426 09:57:41.873940  1683 ts_itest-base.cc:113] --------------
I20240426 09:57:41.874006  1683 ts_itest-base.cc:114] 3 tablet servers
I20240426 09:57:41.874056  1683 ts_itest-base.cc:115] 3 replicas per TS
I20240426 09:57:41.874118  1683 ts_itest-base.cc:116] --------------
2024-04-26T09:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-26T09:57:41Z Disabled control of system clock
I20240426 09:57:41.899204  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.164.254:41179
--webserver_interface=127.1.164.254
--webserver_port=0
--builtin_ntp_servers=127.1.164.212:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.164.254:41179
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240426 09:57:42.229017  1695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:57:42.229351  1695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:57:42.229455  1695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:57:42.237177  1695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240426 09:57:42.237301  1695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:57:42.237345  1695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240426 09:57:42.237402  1695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240426 09:57:42.246788  1695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.164.254:41179
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.164.254:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.1.164.254
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:57:42.248184  1695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:57:42.250000  1695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:42.261551  1701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:42.261664  1700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:42.261574  1703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:42.261934  1695 server_base.cc:1034] running on GCE node
I20240426 09:57:42.263008  1695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:57:42.264549  1695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:57:42.265816  1695 hybrid_clock.cc:648] HybridClock initialized: now 1714125462265720 us; error 74 us; skew 500 ppm
I20240426 09:57:42.266206  1695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:42.268936  1695 webserver.cc:460] Webserver started at http://127.1.164.254:43351/ using document root <none> and password file <none>
I20240426 09:57:42.269773  1695 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:42.269888  1695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:42.270222  1695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:42.273082  1695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/instance:
uuid: "6b719224e4564e75913cae085a075703"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.273856  1695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal/instance:
uuid: "6b719224e4564e75913cae085a075703"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.279503  1695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240426 09:57:42.283295  1709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:42.284752  1695 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240426 09:57:42.285003  1695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
uuid: "6b719224e4564e75913cae085a075703"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.285197  1695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:42.320683  1695 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:42.321586  1695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:57:42.321842  1695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:42.339855  1695 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.254:41179
I20240426 09:57:42.339885  1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.254:41179 every 8 connection(s)
I20240426 09:57:42.341292  1695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
I20240426 09:57:42.345103  1761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:42.346884  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 1695
I20240426 09:57:42.347124  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal/instance
I20240426 09:57:42.355818  1761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703: Bootstrap starting.
I20240426 09:57:42.358815  1761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:42.359817  1761 log.cc:824] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703: Log is configured to *not* fsync() on all Append() calls
I20240426 09:57:42.362573  1761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703: No bootstrap required, opened a new log
I20240426 09:57:42.369051  1761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } }
I20240426 09:57:42.369452  1761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:42.369604  1761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b719224e4564e75913cae085a075703, State: Initialized, Role: FOLLOWER
I20240426 09:57:42.370378  1761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } }
I20240426 09:57:42.370661  1761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:57:42.370795  1761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:57:42.370980  1761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:42.373507  1761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } }
I20240426 09:57:42.373998  1761 leader_election.cc:302] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b719224e4564e75913cae085a075703; no voters: 
I20240426 09:57:42.374572  1761 leader_election.cc:288] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 09:57:42.374713  1766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:42.375802  1766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [term 1 LEADER]: Becoming Leader. State: Replica: 6b719224e4564e75913cae085a075703, State: Running, Role: LEADER
I20240426 09:57:42.376318  1766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } }
I20240426 09:57:42.376629  1761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:57:42.379264  1768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b719224e4564e75913cae085a075703. Latest consensus state: current_term: 1 leader_uuid: "6b719224e4564e75913cae085a075703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } } }
I20240426 09:57:42.379676  1768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [sys.catalog]: This master's current role is: LEADER
I20240426 09:57:42.380295  1767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b719224e4564e75913cae085a075703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b719224e4564e75913cae085a075703" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 41179 } } }
I20240426 09:57:42.380637  1767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703 [sys.catalog]: This master's current role is: LEADER
I20240426 09:57:42.382014  1772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:57:42.388011  1772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:57:42.397598  1772 catalog_manager.cc:1347] Generated new cluster ID: 40904357dd3d4e1faeaaecd5c9b3482c
I20240426 09:57:42.397727  1772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:57:42.413894  1772 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:57:42.415118  1772 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:57:42.426680  1772 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6b719224e4564e75913cae085a075703: Generated new TSK 0
I20240426 09:57:42.427587  1772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:57:42.435200  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.193:0
--local_ip_for_outbound_sockets=127.1.164.193
--webserver_interface=127.1.164.193
--webserver_port=0
--tserver_master_addrs=127.1.164.254:41179
--builtin_ntp_servers=127.1.164.212:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240426 09:57:42.797816  1785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:57:42.798116  1785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:57:42.798312  1785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:57:42.805910  1785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:57:42.806061  1785 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:57:42.806135  1785 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:57:42.806216  1785 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:57:42.806337  1785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.193
I20240426 09:57:42.815605  1785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.1.164.193
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.164.254:41179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.193
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:57:42.817090  1785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:57:42.818853  1785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:42.830812  1791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:42.830757  1793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:42.830899  1790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:42.831521  1785 server_base.cc:1034] running on GCE node
I20240426 09:57:42.832120  1785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:57:42.833115  1785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:57:42.834358  1785 hybrid_clock.cc:648] HybridClock initialized: now 1714125462834302 us; error 55 us; skew 500 ppm
I20240426 09:57:42.834702  1785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:42.837042  1785 webserver.cc:460] Webserver started at http://127.1.164.193:42697/ using document root <none> and password file <none>
I20240426 09:57:42.837812  1785 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:42.837926  1785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:42.838258  1785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:42.840945  1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/instance:
uuid: "56e7c87ce920461cba4d6cccc793ebd4"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.841724  1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal/instance:
uuid: "56e7c87ce920461cba4d6cccc793ebd4"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.847330  1785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240426 09:57:42.851035  1799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:42.852423  1785 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240426 09:57:42.852594  1785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
uuid: "56e7c87ce920461cba4d6cccc793ebd4"
format_stamp: "Formatted at 2024-04-26 09:57:42 on dist-test-slave-8sfd"
I20240426 09:57:42.852814  1785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:42.876636  1785 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:42.877625  1785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:57:42.877918  1785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:42.878883  1785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:57:42.880439  1785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:57:42.880532  1785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:42.880649  1785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:57:42.880729  1785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:42.912210  1785 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.193:38345
I20240426 09:57:42.912258  1911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.193:38345 every 8 connection(s)
I20240426 09:57:42.913600  1785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
I20240426 09:57:42.915181  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 1785
I20240426 09:57:42.915411  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal/instance
I20240426 09:57:42.919206  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.194:0
--local_ip_for_outbound_sockets=127.1.164.194
--webserver_interface=127.1.164.194
--webserver_port=0
--tserver_master_addrs=127.1.164.254:41179
--builtin_ntp_servers=127.1.164.212:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 09:57:42.929513  1912 heartbeater.cc:347] Connected to a master server at 127.1.164.254:41179
I20240426 09:57:42.929874  1912 heartbeater.cc:464] Registering TS with master...
I20240426 09:57:42.931000  1912 heartbeater.cc:511] Master 127.1.164.254:41179 requested a full tablet report, sending...
I20240426 09:57:42.933506  1726 ts_manager.cc:194] Registered new tserver with Master: 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345)
I20240426 09:57:42.934996  1726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.193:45623
W20240426 09:57:43.258759  1916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:57:43.259048  1916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:57:43.259235  1916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:57:43.266870  1916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:57:43.267030  1916 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:57:43.267099  1916 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:57:43.267267  1916 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:57:43.267442  1916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.194
I20240426 09:57:43.277077  1916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.1.164.194
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.164.254:41179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.194
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:57:43.278487  1916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:57:43.280452  1916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:43.293696  1922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:43.293650  1924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:43.293720  1916 server_base.cc:1034] running on GCE node
W20240426 09:57:43.294021  1921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:43.294701  1916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:57:43.295734  1916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:57:43.296937  1916 hybrid_clock.cc:648] HybridClock initialized: now 1714125463296897 us; error 52 us; skew 500 ppm
I20240426 09:57:43.297281  1916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:43.299726  1916 webserver.cc:460] Webserver started at http://127.1.164.194:45523/ using document root <none> and password file <none>
I20240426 09:57:43.300511  1916 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:43.300629  1916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:43.301019  1916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:43.303781  1916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/instance:
uuid: "7a0fdab05e5f434d8c1e538483da228e"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.304553  1916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal/instance:
uuid: "7a0fdab05e5f434d8c1e538483da228e"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.310631  1916 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.002s
I20240426 09:57:43.314257  1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.315734  1916 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240426 09:57:43.315923  1916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
uuid: "7a0fdab05e5f434d8c1e538483da228e"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.316141  1916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:43.336961  1916 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:43.337869  1916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:57:43.338169  1916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:43.339221  1916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:57:43.340945  1916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:57:43.341053  1916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.341171  1916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:57:43.341253  1916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.375331  1916 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.194:35203
I20240426 09:57:43.375404  2042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.194:35203 every 8 connection(s)
I20240426 09:57:43.376714  1916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
I20240426 09:57:43.380913  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 1916
I20240426 09:57:43.381114  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal/instance
I20240426 09:57:43.384420  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.195:0
--local_ip_for_outbound_sockets=127.1.164.195
--webserver_interface=127.1.164.195
--webserver_port=0
--tserver_master_addrs=127.1.164.254:41179
--builtin_ntp_servers=127.1.164.212:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240426 09:57:43.390972  2043 heartbeater.cc:347] Connected to a master server at 127.1.164.254:41179
I20240426 09:57:43.391317  2043 heartbeater.cc:464] Registering TS with master...
I20240426 09:57:43.392233  2043 heartbeater.cc:511] Master 127.1.164.254:41179 requested a full tablet report, sending...
I20240426 09:57:43.393919  1726 ts_manager.cc:194] Registered new tserver with Master: 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203)
I20240426 09:57:43.394752  1726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.194:41537
W20240426 09:57:43.722081  2046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:57:43.722384  2046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:57:43.722563  2046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:57:43.730229  2046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:57:43.730388  2046 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240426 09:57:43.730471  2046 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240426 09:57:43.730563  2046 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240426 09:57:43.730680  2046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.195
I20240426 09:57:43.740125  2046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.1.164.195
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.1.164.254:41179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.195
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:57:43.741643  2046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:57:43.743436  2046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:43.755846  2052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:43.755882  2051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:43.755988  2054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:43.757560  2046 server_base.cc:1034] running on GCE node
I20240426 09:57:43.758304  2046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:57:43.759339  2046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:57:43.760550  2046 hybrid_clock.cc:648] HybridClock initialized: now 1714125463760505 us; error 44 us; skew 500 ppm
I20240426 09:57:43.760922  2046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:43.767339  2046 webserver.cc:460] Webserver started at http://127.1.164.195:36699/ using document root <none> and password file <none>
I20240426 09:57:43.768205  2046 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:43.768323  2046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:43.768644  2046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:43.771378  2046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/instance:
uuid: "57c0b8293f214a6ba5bec482f225d3b3"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.772109  2046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal/instance:
uuid: "57c0b8293f214a6ba5bec482f225d3b3"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.777582  2046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240426 09:57:43.781246  2060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.782812  2046 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240426 09:57:43.782977  2046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
uuid: "57c0b8293f214a6ba5bec482f225d3b3"
format_stamp: "Formatted at 2024-04-26 09:57:43 on dist-test-slave-8sfd"
I20240426 09:57:43.783212  2046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:43.820178  2046 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:43.821066  2046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:57:43.821377  2046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:43.822352  2046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:57:43.823824  2046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:57:43.823920  2046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.824026  2046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:57:43.824097  2046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:43.856105  2046 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.195:33183
I20240426 09:57:43.856139  2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.195:33183 every 8 connection(s)
I20240426 09:57:43.857523  2046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
I20240426 09:57:43.863862  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 2046
I20240426 09:57:43.864059  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal/instance
I20240426 09:57:43.870729  2173 heartbeater.cc:347] Connected to a master server at 127.1.164.254:41179
I20240426 09:57:43.871059  2173 heartbeater.cc:464] Registering TS with master...
I20240426 09:57:43.871958  2173 heartbeater.cc:511] Master 127.1.164.254:41179 requested a full tablet report, sending...
I20240426 09:57:43.873576  1726 ts_manager.cc:194] Registered new tserver with Master: 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:43.874442  1726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.195:42149
I20240426 09:57:43.880854  1683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:57:43.903311  1726 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49196:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240426 09:57:43.909418  1726 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:57:43.938839  1912 heartbeater.cc:503] Master 127.1.164.254:41179 was elected leader, sending a full tablet report...
I20240426 09:57:43.955328  1845 tablet_service.cc:1467] Processing CreateTablet for tablet 5a863278d07243499fe30d5c68ed0b41 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:57:43.956058  1846 tablet_service.cc:1467] Processing CreateTablet for tablet bbe61252956748109d8ca9a492c04e24 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:57:43.955704  1847 tablet_service.cc:1467] Processing CreateTablet for tablet b746c53e16154116b5ee848f02464846 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:57:43.958904  1847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b746c53e16154116b5ee848f02464846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.959852  1846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe61252956748109d8ca9a492c04e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.961122  1845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a863278d07243499fe30d5c68ed0b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.960287  1976 tablet_service.cc:1467] Processing CreateTablet for tablet 5a863278d07243499fe30d5c68ed0b41 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:57:43.961930  1977 tablet_service.cc:1467] Processing CreateTablet for tablet bbe61252956748109d8ca9a492c04e24 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:57:43.962350  1976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a863278d07243499fe30d5c68ed0b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.960494  1978 tablet_service.cc:1467] Processing CreateTablet for tablet b746c53e16154116b5ee848f02464846 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:57:43.962774  1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe61252956748109d8ca9a492c04e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.964205  1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b746c53e16154116b5ee848f02464846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.964305  2108 tablet_service.cc:1467] Processing CreateTablet for tablet b746c53e16154116b5ee848f02464846 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240426 09:57:43.966239  2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b746c53e16154116b5ee848f02464846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.970436  2107 tablet_service.cc:1467] Processing CreateTablet for tablet bbe61252956748109d8ca9a492c04e24 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240426 09:57:43.970965  2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe61252956748109d8ca9a492c04e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.972081  2106 tablet_service.cc:1467] Processing CreateTablet for tablet 5a863278d07243499fe30d5c68ed0b41 (DEFAULT_TABLE table=TestTable [id=de777f58bb8a4686a04e568255888d71]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240426 09:57:43.972622  2106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a863278d07243499fe30d5c68ed0b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:43.983392  2191 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Bootstrap starting.
I20240426 09:57:43.983989  2192 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Bootstrap starting.
I20240426 09:57:43.987700  2191 tablet_bootstrap.cc:654] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:43.988250  2192 tablet_bootstrap.cc:654] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:43.989064  2191 log.cc:824] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Log is configured to *not* fsync() on all Append() calls
I20240426 09:57:43.989864  2192 log.cc:824] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Log is configured to *not* fsync() on all Append() calls
I20240426 09:57:43.994130  2195 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Bootstrap starting.
I20240426 09:57:44.001042  2195 tablet_bootstrap.cc:654] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.001665  2192 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: No bootstrap required, opened a new log
I20240426 09:57:44.001973  2192 ts_tablet_manager.cc:1367] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.010s
I20240426 09:57:44.002938  2195 log.cc:824] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Log is configured to *not* fsync() on all Append() calls
I20240426 09:57:44.003878  2191 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: No bootstrap required, opened a new log
I20240426 09:57:44.004163  2191 ts_tablet_manager.cc:1367] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.011s
I20240426 09:57:44.005527  2195 tablet_bootstrap.cc:492] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: No bootstrap required, opened a new log
I20240426 09:57:44.005816  2195 ts_tablet_manager.cc:1367] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.004s
I20240426 09:57:44.011180  2192 raft_consensus.cc:348] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.011677  2192 raft_consensus.cc:374] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.011780  2192 raft_consensus.cc:729] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.012300  2195 raft_consensus.cc:348] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.012653  2195 raft_consensus.cc:374] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.012779  2195 raft_consensus.cc:729] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c0b8293f214a6ba5bec482f225d3b3, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.012645  2192 consensus_queue.cc:260] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.013448  2195 consensus_queue.cc:260] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.014647  2173 heartbeater.cc:503] Master 127.1.164.254:41179 was elected leader, sending a full tablet report...
I20240426 09:57:44.015126  2195 ts_tablet_manager.cc:1398] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20240426 09:57:44.015537  2195 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3: Bootstrap starting.
I20240426 09:57:44.017208  2195 tablet_bootstrap.cc:654] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.014336  2191 raft_consensus.cc:348] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.018337  2191 raft_consensus.cc:374] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.018481  2191 raft_consensus.cc:729] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e7c87ce920461cba4d6cccc793ebd4, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.019268  2191 consensus_queue.cc:260] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.019801  2195 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3: No bootstrap required, opened a new log
I20240426 09:57:44.020068  2195 ts_tablet_manager.cc:1367] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 09:57:44.021632  2195 raft_consensus.cc:348] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.022189  2195 raft_consensus.cc:374] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.022387  2195 raft_consensus.cc:729] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c0b8293f214a6ba5bec482f225d3b3, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.022730  2195 consensus_queue.cc:260] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.023540  2195 ts_tablet_manager.cc:1398] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240426 09:57:44.023873  2195 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3: Bootstrap starting.
I20240426 09:57:44.024626  2191 ts_tablet_manager.cc:1398] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.006s
I20240426 09:57:44.025125  2191 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4: Bootstrap starting.
I20240426 09:57:44.026412  2195 tablet_bootstrap.cc:654] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.026914  2191 tablet_bootstrap.cc:654] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.026932  2192 ts_tablet_manager.cc:1398] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Time spent starting tablet: real 0.025s	user 0.013s	sys 0.009s
I20240426 09:57:44.028209  2043 heartbeater.cc:503] Master 127.1.164.254:41179 was elected leader, sending a full tablet report...
I20240426 09:57:44.028332  2192 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e: Bootstrap starting.
I20240426 09:57:44.030191  2192 tablet_bootstrap.cc:654] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.032383  2195 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3: No bootstrap required, opened a new log
I20240426 09:57:44.032567  2195 ts_tablet_manager.cc:1367] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.003s
I20240426 09:57:44.032878  2192 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e: No bootstrap required, opened a new log
I20240426 09:57:44.033138  2192 ts_tablet_manager.cc:1367] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240426 09:57:44.033440  2195 raft_consensus.cc:348] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.033705  2195 raft_consensus.cc:374] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.033792  2195 raft_consensus.cc:729] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c0b8293f214a6ba5bec482f225d3b3, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.034082  2195 consensus_queue.cc:260] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.034679  2195 ts_tablet_manager.cc:1398] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240426 09:57:44.035602  2192 raft_consensus.cc:348] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.035856  2192 raft_consensus.cc:374] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.036000  2192 raft_consensus.cc:729] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.036080  2191 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4: No bootstrap required, opened a new log
I20240426 09:57:44.036250  2191 ts_tablet_manager.cc:1367] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.008s
I20240426 09:57:44.036291  2192 consensus_queue.cc:260] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.037050  2192 ts_tablet_manager.cc:1398] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240426 09:57:44.037186  2191 raft_consensus.cc:348] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.037408  2191 raft_consensus.cc:374] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.037513  2191 raft_consensus.cc:729] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e7c87ce920461cba4d6cccc793ebd4, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.037801  2191 consensus_queue.cc:260] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.037976  2192 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e: Bootstrap starting.
I20240426 09:57:44.038317  2191 ts_tablet_manager.cc:1398] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240426 09:57:44.038651  2191 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4: Bootstrap starting.
I20240426 09:57:44.039613  2192 tablet_bootstrap.cc:654] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.040293  2191 tablet_bootstrap.cc:654] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:44.043114  2192 tablet_bootstrap.cc:492] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e: No bootstrap required, opened a new log
I20240426 09:57:44.043285  2192 ts_tablet_manager.cc:1367] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240426 09:57:44.043627  2191 tablet_bootstrap.cc:492] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4: No bootstrap required, opened a new log
I20240426 09:57:44.043784  2191 ts_tablet_manager.cc:1367] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240426 09:57:44.044082  2192 raft_consensus.cc:348] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.044325  2192 raft_consensus.cc:374] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.044407  2192 raft_consensus.cc:729] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.044481  2191 raft_consensus.cc:348] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.044739  2191 raft_consensus.cc:374] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:44.044857  2191 raft_consensus.cc:729] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e7c87ce920461cba4d6cccc793ebd4, State: Initialized, Role: FOLLOWER
I20240426 09:57:44.044713  2192 consensus_queue.cc:260] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.045110  2191 consensus_queue.cc:260] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.045315  2192 ts_tablet_manager.cc:1398] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240426 09:57:44.045677  2191 ts_tablet_manager.cc:1398] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240426 09:57:44.099970  2197 raft_consensus.cc:482] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:57:44.100253  2197 raft_consensus.cc:504] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.102105  2197 leader_election.cc:288] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
W20240426 09:57:44.109263  2174 tablet.cc:2369] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:57:44.109908  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a863278d07243499fe30d5c68ed0b41" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4" is_pre_election: true
I20240426 09:57:44.110486  1867 raft_consensus.cc:2454] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 0.
I20240426 09:57:44.110523  2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a863278d07243499fe30d5c68ed0b41" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c0b8293f214a6ba5bec482f225d3b3" is_pre_election: true
I20240426 09:57:44.110981  2128 raft_consensus.cc:2454] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 0.
I20240426 09:57:44.111135  1931 leader_election.cc:302] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 
I20240426 09:57:44.111531  2197 raft_consensus.cc:2781] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:57:44.111666  2197 raft_consensus.cc:482] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:57:44.111763  2197 raft_consensus.cc:3037] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.113874  2197 raft_consensus.cc:504] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.114441  2197 leader_election.cc:288] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 1 election: Requested vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:44.114938  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a863278d07243499fe30d5c68ed0b41" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4"
I20240426 09:57:44.115000  2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a863278d07243499fe30d5c68ed0b41" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c0b8293f214a6ba5bec482f225d3b3"
I20240426 09:57:44.115111  1867 raft_consensus.cc:3037] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.115204  2128 raft_consensus.cc:3037] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.117339  1867 raft_consensus.cc:2454] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 1.
I20240426 09:57:44.117419  2128 raft_consensus.cc:2454] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 1.
I20240426 09:57:44.117753  1931 leader_election.cc:302] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 
I20240426 09:57:44.118052  2197 raft_consensus.cc:2781] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:44.118527  2197 raft_consensus.cc:686] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [term 1 LEADER]: Becoming Leader. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Running, Role: LEADER
I20240426 09:57:44.119025  2197 consensus_queue.cc:237] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.123734  1724 catalog_manager.cc:5484] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194). New cstate: current_term: 1 leader_uuid: "7a0fdab05e5f434d8c1e538483da228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } health_report { overall_health: UNKNOWN } } }
W20240426 09:57:44.128901  2044 tablet.cc:2369] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:57:44.156250  2198 raft_consensus.cc:482] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:57:44.156448  2198 raft_consensus.cc:504] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.158229  2198 leader_election.cc:288] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203)
I20240426 09:57:44.165010  1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e" is_pre_election: true
I20240426 09:57:44.165294  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4" is_pre_election: true
I20240426 09:57:44.165426  1998 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 0.
I20240426 09:57:44.165545  1867 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 0.
I20240426 09:57:44.166021  2063 leader_election.cc:302] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c0b8293f214a6ba5bec482f225d3b3, 7a0fdab05e5f434d8c1e538483da228e; no voters: 
W20240426 09:57:44.166131  1913 tablet.cc:2369] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:57:44.166419  2198 raft_consensus.cc:2781] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:57:44.166553  2198 raft_consensus.cc:482] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:57:44.166652  2198 raft_consensus.cc:3037] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.168993  2198 raft_consensus.cc:504] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.169652  2198 leader_election.cc:288] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 election: Requested vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203)
I20240426 09:57:44.170001  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4"
I20240426 09:57:44.170202  1867 raft_consensus.cc:3037] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.170161  1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e"
I20240426 09:57:44.170351  1998 raft_consensus.cc:3037] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.172122  1867 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 1.
I20240426 09:57:44.172199  2198 raft_consensus.cc:482] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:57:44.172330  2198 raft_consensus.cc:504] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.172545  1998 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 1.
I20240426 09:57:44.172626  2061 leader_election.cc:302] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 57c0b8293f214a6ba5bec482f225d3b3; no voters: 
I20240426 09:57:44.173020  2198 leader_election.cc:288] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203)
I20240426 09:57:44.173192  2198 raft_consensus.cc:2781] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:44.173355  2198 raft_consensus.cc:686] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 57c0b8293f214a6ba5bec482f225d3b3, State: Running, Role: LEADER
I20240426 09:57:44.173591  1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e" is_pre_election: true
I20240426 09:57:44.173897  1998 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 0.
I20240426 09:57:44.173823  2198 consensus_queue.cc:237] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.174342  2063 leader_election.cc:302] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c0b8293f214a6ba5bec482f225d3b3, 7a0fdab05e5f434d8c1e538483da228e; no voters: 
I20240426 09:57:44.177495  2198 raft_consensus.cc:2781] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:57:44.177613  2198 raft_consensus.cc:482] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:57:44.177728  2198 raft_consensus.cc:3037] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.179754  2198 raft_consensus.cc:504] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.180297  2198 leader_election.cc:288] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 election: Requested vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203)
I20240426 09:57:44.181502  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4" is_pre_election: true
I20240426 09:57:44.181746  1867 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 0.
I20240426 09:57:44.181646  1866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4"
I20240426 09:57:44.181967  1866 raft_consensus.cc:3037] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.182241  1724 catalog_manager.cc:5484] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195). New cstate: current_term: 1 leader_uuid: "57c0b8293f214a6ba5bec482f225d3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } health_report { overall_health: HEALTHY } } }
I20240426 09:57:44.184373  1866 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 1.
I20240426 09:57:44.184839  2061 leader_election.cc:302] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 57c0b8293f214a6ba5bec482f225d3b3; no voters: 
I20240426 09:57:44.185277  1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "57c0b8293f214a6ba5bec482f225d3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e"
I20240426 09:57:44.185528  1998 raft_consensus.cc:3037] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:44.185683  2212 raft_consensus.cc:2781] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:44.185855  2212 raft_consensus.cc:686] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 57c0b8293f214a6ba5bec482f225d3b3, State: Running, Role: LEADER
I20240426 09:57:44.186223  2212 consensus_queue.cc:237] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:44.188359  1998 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c0b8293f214a6ba5bec482f225d3b3 in term 1.
I20240426 09:57:44.190057  1724 catalog_manager.cc:5484] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195). New cstate: current_term: 1 leader_uuid: "57c0b8293f214a6ba5bec482f225d3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } health_report { overall_health: HEALTHY } } }
I20240426 09:57:44.228387  1683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:57:44.231495  1683 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 56e7c87ce920461cba4d6cccc793ebd4 to finish bootstrapping
I20240426 09:57:44.241406  1683 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 7a0fdab05e5f434d8c1e538483da228e to finish bootstrapping
I20240426 09:57:44.249274  1683 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 57c0b8293f214a6ba5bec482f225d3b3 to finish bootstrapping
I20240426 09:57:44.258314  2216 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
  http://127.1.164.254:43351/metrics
  http://127.1.164.193:42697/threadz
  http://127.1.164.254:43351/startup
  http://127.1.164.193:42697/threadz?group=all
  http://127.1.164.195:36699/scans
  http://127.1.164.194:45523/scans
  http://127.1.164.193:42697/scans
  http://127.1.164.193:42697/mem-trackers
  http://127.1.164.195:36699/startup
  http://127.1.164.194:45523/threadz?group=all
  http://127.1.164.193:42697/metrics
  http://127.1.164.254:43351/dump-entities
  http://127.1.164.254:43351/stacks
  http://127.1.164.254:43351/tables
  http://127.1.164.195:36699/threadz
  http://127.1.164.195:36699/metrics
  http://127.1.164.195:36699/tablets
  http://127.1.164.193:42697/stacks
  http://127.1.164.194:45523/metrics
  http://127.1.164.193:42697/transactions?tablet_id=5a863278d07243499fe30d5c68ed0b41
  http://127.1.164.195:36699/mem-trackers
  http://127.1.164.193:42697/startup
  http://127.1.164.194:45523/maintenance-manager
  http://127.1.164.254:43351/threadz?group=all
  http://127.1.164.254:43351/ipki-ca-cert-pem
  http://127.1.164.254:43351/ipki-ca-cert-der
  http://127.1.164.254:43351/masters
  http://127.1.164.194:45523/threadz
  http://127.1.164.195:36699/stacks
  http://127.1.164.195:36699/transactions?tablet_id=5a863278d07243499fe30d5c68ed0b41
  http://127.1.164.194:45523/stacks
  http://127.1.164.254:43351/ipki-ca-cert
  http://127.1.164.254:43351/tablet-servers
  http://127.1.164.254:43351/threadz
  http://127.1.164.193:42697/tablets
  http://127.1.164.254:43351/mem-trackers
  http://127.1.164.194:45523/transactions?tablet_id=5a863278d07243499fe30d5c68ed0b41
  http://127.1.164.193:42697/maintenance-manager
  http://127.1.164.195:36699/threadz?group=all
  http://127.1.164.194:45523/startup
  http://127.1.164.195:36699/maintenance-manager
  http://127.1.164.194:45523/mem-trackers
  http://127.1.164.194:45523/tablets
I20240426 09:57:44.263576  1683 linked_list-test-util.h:401] Running for: 30.000s
I20240426 09:57:44.263748  1683 linked_list-test-util.h:402] Sampling every 10000000 us
I20240426 09:57:44.263854  1683 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240426 09:57:44.298017  1866 raft_consensus.cc:1261] T 5a863278d07243499fe30d5c68ed0b41 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Refusing update from remote peer 7a0fdab05e5f434d8c1e538483da228e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.298242  2128 raft_consensus.cc:1261] T 5a863278d07243499fe30d5c68ed0b41 P 57c0b8293f214a6ba5bec482f225d3b3 [term 1 FOLLOWER]: Refusing update from remote peer 7a0fdab05e5f434d8c1e538483da228e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.299326  2197 consensus_queue.cc:1035] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.300669  2208 consensus_queue.cc:1035] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.309983  1867 raft_consensus.cc:1261] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Refusing update from remote peer 57c0b8293f214a6ba5bec482f225d3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.310319  1998 raft_consensus.cc:1261] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Refusing update from remote peer 57c0b8293f214a6ba5bec482f225d3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.311084  2211 consensus_queue.cc:1035] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.311498  2198 consensus_queue.cc:1035] T bbe61252956748109d8ca9a492c04e24 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.314260  1997 raft_consensus.cc:1261] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Refusing update from remote peer 57c0b8293f214a6ba5bec482f225d3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.315083  2212 consensus_queue.cc:1035] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.329854  1865 raft_consensus.cc:1261] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Refusing update from remote peer 57c0b8293f214a6ba5bec482f225d3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 09:57:44.331357  2212 consensus_queue.cc:1035] T b746c53e16154116b5ee848f02464846 P 57c0b8293f214a6ba5bec482f225d3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 09:57:44.354533  2225 mvcc.cc:205] Tried to move back new op lower bound from 7021057901799034880 to 7021057901309894656. Current Snapshot: MvccSnapshot[applied={T|T < 7021057901799034880}]
I20240426 09:57:44.373806  2224 mvcc.cc:205] Tried to move back new op lower bound from 7021057901750878208 to 7021057901035511808. Current Snapshot: MvccSnapshot[applied={T|T < 7021057901750878208}]
I20240426 09:57:44.416329  2231 mvcc.cc:205] Tried to move back new op lower bound from 7021057901799034880 to 7021057901309894656. Current Snapshot: MvccSnapshot[applied={T|T < 7021057901799034880}]
I20240426 09:57:44.421545  2236 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:44.456034  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:44.460064  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240426 09:57:47.281772  2236 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7f10acf3cdbd  google::LogMessage::Fail() at ??:0
    @     0x7f10acf40b83  google::LogMessage::SendToLog() at ??:0
    @     0x7f10acf3c7bc  google::LogMessage::Flush() at ??:0
    @     0x7f10acf3df49  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f10afb64880  kudu::operator-() at ??:0
    @     0x7f10bc7d639a  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f10bc7a082e  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f10bc7a0586  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f10bc74d529  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f10bf2170a7  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7f10bf225c5c  kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7f10bf225ac2  std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f10b8c666aa  std::function<>::operator()() at ??:0
    @     0x7f10b8c5ee66  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f10b8c59e71  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f10b8c57fa7  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f10b8c95709  handle_request at ??:0
    @     0x7f10b8c97deb  worker_thread at ??:0
    @     0x7f10b2fad6db  start_thread at ??:0
    @     0x7f10abdef71f  clone at ??:0
W20240426 09:57:47.527408  2176 connection.cc:535] client connection to 127.1.164.195:33183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240426 09:57:47.527746  2176 meta_cache.cc:302] tablet b746c53e16154116b5ee848f02464846: replica 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240426 09:57:47.527897  1931 connection.cc:535] client connection to 127.1.164.195:33183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240426 09:57:47.528120  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240426 09:57:47.538463  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:57:47.547267  1827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.547220  1826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.547330  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.552290  1958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.554688  1958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.556046  1958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.571497  1827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.571648  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.571586  1826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.579955  1958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.582101  1958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.582427  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.593792  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.602619  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.607031  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.614430  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.617563  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.635670  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.647209  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.650468  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.658314  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.660171  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.686021  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.692248  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.703881  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.712352  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.738997  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.753546  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.760133  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.772378  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.777805  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.802485  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.825004  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.828338  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.843173  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.848423  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.876379  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.898986  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.905400  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.926746  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:47.962576  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.987533  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:47.993359  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.009884  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.020570  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.058358  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.089182  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.090629  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.107672  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.123096  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
I20240426 09:57:48.127668  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:48.129045  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:57:48.161734  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.194227  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.197399  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.232360  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.306628  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.309692  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.331125  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.345983  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.348055  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.371095  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.428421  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.430193  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.469265  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.474664  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.497740  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.543529  2176 meta_cache.cc:302] tablet bbe61252956748109d8ca9a492c04e24: replica 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183) has failed: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
W20240426 09:57:48.559033  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.564617  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.589733  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.604058  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.611671  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.702114  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.713830  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.734372  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.753007  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.763620  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
I20240426 09:57:48.772046  2291 raft_consensus.cc:482] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
I20240426 09:57:48.772261  2291 raft_consensus.cc:504] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.773905  2291 leader_election.cc:288] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:48.777827  2288 raft_consensus.cc:482] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
I20240426 09:57:48.778088  2288 raft_consensus.cc:504] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.778988  2288 leader_election.cc:288] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
W20240426 09:57:48.781056  1931 leader_election.cc:334] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.787629  1866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4" is_pre_election: true
I20240426 09:57:48.787984  1866 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 1.
I20240426 09:57:48.788570  1931 leader_election.cc:302] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 57c0b8293f214a6ba5bec482f225d3b3
I20240426 09:57:48.788909  2288 raft_consensus.cc:2781] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:57:48.789057  2288 raft_consensus.cc:482] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
W20240426 09:57:48.789151  1957 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40612: Illegal state: replica 7a0fdab05e5f434d8c1e538483da228e is not leader of this config: current role FOLLOWER
W20240426 09:57:48.789207  1800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.789175  2288 raft_consensus.cc:3037] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:57:48.793015  2291 raft_consensus.cc:482] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
I20240426 09:57:48.793334  2291 raft_consensus.cc:504] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.794469  2291 leader_election.cc:288] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194:35203), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:48.795475  2288 raft_consensus.cc:504] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.796494  2288 leader_election.cc:288] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: Requested vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:48.796936  2288 raft_consensus.cc:482] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
I20240426 09:57:48.797138  2288 raft_consensus.cc:504] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.797762  2288 leader_election.cc:288] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
W20240426 09:57:48.798362  1931 leader_election.cc:334] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
W20240426 09:57:48.799499  1931 leader_election.cc:334] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.801425  1867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4" is_pre_election: true
I20240426 09:57:48.801687  1867 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 1.
I20240426 09:57:48.801960  1866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4"
W20240426 09:57:48.802024  1800 leader_election.cc:334] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.802140  1866 raft_consensus.cc:3037] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:57:48.802556  1931 leader_election.cc:302] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 57c0b8293f214a6ba5bec482f225d3b3
I20240426 09:57:48.803040  2296 raft_consensus.cc:2781] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 09:57:48.803213  2296 raft_consensus.cc:482] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57c0b8293f214a6ba5bec482f225d3b3)
I20240426 09:57:48.803299  1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b746c53e16154116b5ee848f02464846" candidate_uuid: "56e7c87ce920461cba4d6cccc793ebd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e" is_pre_election: true
I20240426 09:57:48.803397  2296 raft_consensus.cc:3037] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:57:48.803457  1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "56e7c87ce920461cba4d6cccc793ebd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "7a0fdab05e5f434d8c1e538483da228e" is_pre_election: true
I20240426 09:57:48.803617  1998 raft_consensus.cc:2379] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e7c87ce920461cba4d6cccc793ebd4 in current term 2: Already voted for candidate 7a0fdab05e5f434d8c1e538483da228e in this term.
I20240426 09:57:48.804284  1802 leader_election.cc:302] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4; no voters: 57c0b8293f214a6ba5bec482f225d3b3, 7a0fdab05e5f434d8c1e538483da228e
W20240426 09:57:48.804894  1800 leader_election.cc:334] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.806542  2296 raft_consensus.cc:504] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.806857  1866 raft_consensus.cc:2454] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 2.
I20240426 09:57:48.806949  1996 raft_consensus.cc:2379] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e7c87ce920461cba4d6cccc793ebd4 in current term 2: Already voted for candidate 7a0fdab05e5f434d8c1e538483da228e in this term.
I20240426 09:57:48.807281  2296 leader_election.cc:288] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: Requested vote from peers 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345), 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183)
I20240426 09:57:48.807646  2291 raft_consensus.cc:2726] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:57:48.807765  1931 leader_election.cc:302] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 57c0b8293f214a6ba5bec482f225d3b3
I20240426 09:57:48.808182  1802 leader_election.cc:302] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4; no voters: 57c0b8293f214a6ba5bec482f225d3b3, 7a0fdab05e5f434d8c1e538483da228e
I20240426 09:57:48.808207  1866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe61252956748109d8ca9a492c04e24" candidate_uuid: "7a0fdab05e5f434d8c1e538483da228e" candidate_term: 2 candidate_status { last_received { term: 1 index: 132 } } ignore_live_leader: false dest_uuid: "56e7c87ce920461cba4d6cccc793ebd4"
I20240426 09:57:48.808467  1866 raft_consensus.cc:3037] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 09:57:48.808732  2296 raft_consensus.cc:2781] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:57:48.808964  2296 raft_consensus.cc:686] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [term 2 LEADER]: Becoming Leader. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Running, Role: LEADER
I20240426 09:57:48.809306  2296 consensus_queue.cc:237] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 1.132, Last appended by leader: 132, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
W20240426 09:57:48.809669  1931 leader_election.cc:334] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:57:48.812013  1866 raft_consensus.cc:2454] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fdab05e5f434d8c1e538483da228e in term 2.
I20240426 09:57:48.812191  2291 raft_consensus.cc:2726] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:57:48.812831  1931 leader_election.cc:302] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e7c87ce920461cba4d6cccc793ebd4, 7a0fdab05e5f434d8c1e538483da228e; no voters: 57c0b8293f214a6ba5bec482f225d3b3
I20240426 09:57:48.813156  2296 raft_consensus.cc:2781] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 09:57:48.813354  2296 raft_consensus.cc:686] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [term 2 LEADER]: Becoming Leader. State: Replica: 7a0fdab05e5f434d8c1e538483da228e, State: Running, Role: LEADER
I20240426 09:57:48.813169  1719 catalog_manager.cc:5484] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e reported cstate change: term changed from 1 to 2, leader changed from 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195) to 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194). New cstate: current_term: 2 leader_uuid: "7a0fdab05e5f434d8c1e538483da228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } health_report { overall_health: UNKNOWN } } }
I20240426 09:57:48.813654  2296 consensus_queue.cc:237] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 1.132, Last appended by leader: 132, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } }
I20240426 09:57:48.831068  1719 catalog_manager.cc:5484] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e reported cstate change: term changed from 1 to 2, leader changed from 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195) to 7a0fdab05e5f434d8c1e538483da228e (127.1.164.194). New cstate: current_term: 2 leader_uuid: "7a0fdab05e5f434d8c1e538483da228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0fdab05e5f434d8c1e538483da228e" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 35203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c0b8293f214a6ba5bec482f225d3b3" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 33183 } health_report { overall_health: UNKNOWN } } }
W20240426 09:57:48.854107  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
W20240426 09:57:48.867833  1825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54908: Illegal state: replica 56e7c87ce920461cba4d6cccc793ebd4 is not leader of this config: current role FOLLOWER
I20240426 09:57:48.910446  1866 raft_consensus.cc:1261] T bbe61252956748109d8ca9a492c04e24 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Refusing update from remote peer 7a0fdab05e5f434d8c1e538483da228e: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 2 index: 134. (index mismatch)
W20240426 09:57:48.910842  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:57:48.911307  2296 consensus_queue.cc:1035] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20240426 09:57:48.923527  1867 raft_consensus.cc:1261] T b746c53e16154116b5ee848f02464846 P 56e7c87ce920461cba4d6cccc793ebd4 [term 2 FOLLOWER]: Refusing update from remote peer 7a0fdab05e5f434d8c1e538483da228e: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 2 index: 134. (index mismatch)
W20240426 09:57:48.924527  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:57:48.925253  2296 consensus_queue.cc:1035] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7c87ce920461cba4d6cccc793ebd4" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20240426 09:57:49.337479  2217 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/2302/stat: No such file or directory (error 2)
W20240426 09:57:49.884532  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 09:57:50.228847  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:50.230633  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:51.144080  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:57:51.188910  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
W20240426 09:57:51.434682  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:57:51.499501  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 09:57:52.366806  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.222s	user 0.586s	sys 0.113s Metrics: {"bytes_written":149786,"cfile_init":1,"compiler_manager_pool.queue_time_us":101022,"compiler_manager_pool.run_cpu_time_us":692477,"compiler_manager_pool.run_wall_time_us":947356,"dirs.queue_time_us":17041,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":39089,"drs_written":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4,"lbm_write_time_us":27102,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8192,"thread_start_us":33315,"threads_started":2}
I20240426 09:57:52.369786  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 144540 bytes on disk
I20240426 09:57:52.371065  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240426 09:57:52.567976  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240426 09:57:52.571290  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 09:57:52.977934  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.788s	user 0.616s	sys 0.099s Metrics: {"bytes_written":1861,"cfile_init":1,"compiler_manager_pool.queue_time_us":206736,"compiler_manager_pool.run_cpu_time_us":684729,"compiler_manager_pool.run_wall_time_us":1690215,"delete_count":0,"dirs.queue_time_us":7419,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":10745,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3148,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8257,"thread_start_us":25845,"threads_started":2,"update_count":261}
I20240426 09:57:52.980460  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846): perf score=0.014257
I20240426 09:57:53.182701  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.201s	user 0.079s	sys 0.078s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2527,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":7297,"dirs.run_cpu_time_us":427,"dirs.run_wall_time_us":1694,"lbm_read_time_us":538,"lbm_reads_lt_1ms":24,"lbm_write_time_us":468,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":108788,"threads_started":1,"update_count":261}
I20240426 09:57:53.184222  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 145137 bytes on disk
I20240426 09:57:53.187355  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240426 09:57:53.659615  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:53.661453  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:57:53.863157  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 09:57:54.245332  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 09:57:54.283573  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021057942632656896 Inserted count: 18550
W20240426 09:57:55.217761  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 09:57:56.203126  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:56.205271  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:56.383780  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:57:56.450506  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
W20240426 09:57:56.608951  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 09:57:56.797983  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 09:57:56.986516  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.602s	user 0.472s	sys 0.082s Metrics: {"bytes_written":148785,"cfile_init":1,"dirs.queue_time_us":17012,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":927,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2172,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8124,"thread_start_us":16899,"threads_started":1}
I20240426 09:57:56.988261  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014815
W20240426 09:57:57.604070  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240426 09:57:57.830169  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 09:57:57.892218  1804 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.288s	user 0.004s	sys 0.000s
I20240426 09:57:57.921881  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.471s	user 0.602s	sys 0.121s Metrics: {"bytes_written":149443,"cfile_init":1,"dirs.queue_time_us":81856,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":9382,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2882,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8157,"thread_start_us":81752,"threads_started":1}
I20240426 09:57:57.923266  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014814
I20240426 09:57:58.050981  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.061s	user 0.808s	sys 0.055s Metrics: {"bytes_written":1356,"cfile_cache_hit":13,"cfile_cache_hit_bytes":144385,"cfile_cache_miss":59,"cfile_cache_miss_bytes":939079,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":113370,"dirs.run_cpu_time_us":752,"dirs.run_wall_time_us":283728,"drs_written":1,"lbm_read_time_us":3601,"lbm_reads_lt_1ms":91,"lbm_write_time_us":3942,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":529623,"reinsert_count":0,"rows_written":16316,"thread_start_us":138800,"threads_started":7,"update_count":181}
I20240426 09:57:58.052208  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 295363 bytes on disk
I20240426 09:57:58.053267  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240426 09:57:58.054132  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846): perf score=0.005174
I20240426 09:57:58.221369  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.166s	user 0.071s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2008,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":412,"lbm_reads_lt_1ms":20,"lbm_write_time_us":352,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":181}
I20240426 09:57:58.844682  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.915s	user 0.769s	sys 0.015s Metrics: {"bytes_written":304524,"cfile_cache_hit":13,"cfile_cache_hit_bytes":96417,"cfile_cache_miss":63,"cfile_cache_miss_bytes":993911,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":5975,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":43822,"drs_written":1,"lbm_read_time_us":4606,"lbm_reads_lt_1ms":99,"lbm_write_time_us":4007,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":534664,"rows_written":16414,"thread_start_us":5084,"threads_started":7}
I20240426 09:57:58.846385  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 298838 bytes on disk
I20240426 09:57:58.847227  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240426 09:57:59.033593  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 09:57:59.155366  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 09:57:59.375949  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:57:59.382509  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:58:00.548435  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 09:58:01.246970  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240426 09:58:01.729347  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
W20240426 09:58:01.812510  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240426 09:58:01.855216  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:58:01.903702  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:01.905334  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:02.228235  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.499s	user 0.391s	sys 0.028s Metrics: {"bytes_written":143731,"cfile_init":1,"dirs.queue_time_us":6352,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":9861,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1965,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7833,"thread_start_us":6480,"threads_started":1}
I20240426 09:58:02.229781  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014706
I20240426 09:58:02.413763  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.556s	user 0.457s	sys 0.052s Metrics: {"bytes_written":145852,"cfile_init":1,"dirs.queue_time_us":6346,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":8534,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2008,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7957,"thread_start_us":6472,"threads_started":1}
I20240426 09:58:02.414953  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014704
W20240426 09:58:02.879640  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240426 09:58:02.894618  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240426 09:58:03.421959  1931 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.164.193:38345, user_credentials={real_user=slave}} blocked reactor thread for 45816.5us
I20240426 09:58:03.793658  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.563s	user 1.180s	sys 0.034s Metrics: {"bytes_written":2753,"cfile_cache_hit":12,"cfile_cache_hit_bytes":46930,"cfile_cache_miss":78,"cfile_cache_miss_bytes":1584874,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":26956,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1792,"drs_written":1,"lbm_read_time_us":16440,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":113,"lbm_write_time_us":5215,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":791640,"reinsert_count":0,"rows_written":24149,"thread_start_us":26876,"threads_started":1,"update_count":399}
I20240426 09:58:03.798101  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846): perf score=0.007114
I20240426 09:58:03.958119  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.160s	user 0.096s	sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4137,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":74,"lbm_read_time_us":492,"lbm_reads_lt_1ms":24,"lbm_write_time_us":383,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":399}
W20240426 09:58:03.958611  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240426 09:58:03.959048  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 443481 bytes on disk
I20240426 09:58:03.960125  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240426 09:58:04.270767  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021057983506886656 Inserted count: 39650
I20240426 09:58:04.376854  2349 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 11 ops behind the committed index 
W20240426 09:58:04.495585  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240426 09:58:04.814455  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 2.399s	user 1.123s	sys 0.024s Metrics: {"bytes_written":4171,"cfile_cache_hit":13,"cfile_cache_hit_bytes":236385,"cfile_cache_miss":73,"cfile_cache_miss_bytes":1409031,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":61165,"dirs.run_cpu_time_us":1554,"dirs.run_wall_time_us":44596,"drs_written":1,"lbm_read_time_us":4551,"lbm_reads_lt_1ms":105,"lbm_write_time_us":5234,"lbm_writes_lt_1ms":68,"mutex_wait_us":27572,"num_input_rowsets":2,"peak_mem_usage":795122,"reinsert_count":0,"rows_written":24371,"thread_start_us":80976,"threads_started":4,"update_count":621}
I20240426 09:58:04.816367  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 445922 bytes on disk
I20240426 09:58:04.820556  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
I20240426 09:58:04.824530  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846): perf score=0.010667
I20240426 09:58:04.995975  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.163s	user 0.122s	sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":6050,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":507,"lbm_reads_lt_1ms":20,"lbm_write_time_us":375,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":621}
W20240426 09:58:05.234395  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240426 09:58:05.864641  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:05.866447  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:06.500689  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(bbe61252956748109d8ca9a492c04e24): perf score=1.000000
W20240426 09:58:06.570572  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240426 09:58:06.719022  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(bbe61252956748109d8ca9a492c04e24): perf score=1.000000
W20240426 09:58:06.988785  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240426 09:58:07.322150  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.821s	user 0.472s	sys 0.055s Metrics: {"bytes_written":2552,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9840,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":11646,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2243,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7635,"thread_start_us":9993,"threads_started":1,"update_count":369}
I20240426 09:58:07.324365  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(5a863278d07243499fe30d5c68ed0b41): perf score=1.000000
I20240426 09:58:07.455924  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.737s	user 0.449s	sys 0.101s Metrics: {"bytes_written":3012,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":78450,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":2281,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2249,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7706,"thread_start_us":66016,"threads_started":1,"update_count":440}
I20240426 09:58:07.457173  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(5a863278d07243499fe30d5c68ed0b41): perf score=1.000000
I20240426 09:58:07.539458  2368 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 162 ops behind the committed index  [suppressed 5 similar messages]
W20240426 09:58:07.888907  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240426 09:58:07.892940  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240426 09:58:07.949762  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(5a863278d07243499fe30d5c68ed0b41) complete. Timing: real 0.625s	user 0.446s	sys 0.071s Metrics: {"bytes_written":145012,"cfile_init":1,"dirs.queue_time_us":7073,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":9880,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8177,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":7757,"thread_start_us":7180,"threads_started":1}
I20240426 09:58:07.950707  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(5a863278d07243499fe30d5c68ed0b41): 144004 bytes on disk
I20240426 09:58:07.951270  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(5a863278d07243499fe30d5c68ed0b41) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240426 09:58:07.952093  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(bbe61252956748109d8ca9a492c04e24): 139857 bytes on disk
I20240426 09:58:07.952715  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240426 09:58:07.953577  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling MajorDeltaCompactionOp(bbe61252956748109d8ca9a492c04e24): perf score=0.020754
I20240426 09:58:08.036134  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: MajorDeltaCompactionOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.082s	user 0.051s	sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3388,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":847,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":1394,"lbm_read_time_us":331,"lbm_reads_lt_1ms":20,"lbm_write_time_us":379,"lbm_writes_lt_1ms":8,"mutex_wait_us":467,"reinsert_count":0,"thread_start_us":17024,"threads_started":1,"update_count":369}
I20240426 09:58:08.144753  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(5a863278d07243499fe30d5c68ed0b41) complete. Timing: real 0.687s	user 0.471s	sys 0.063s Metrics: {"bytes_written":1315,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6422,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":5958,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2298,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7810,"thread_start_us":6558,"threads_started":1,"update_count":179}
I20240426 09:58:08.146066  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:58:08.287806  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:58:08.909761  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.763s	user 0.534s	sys 0.081s Metrics: {"bytes_written":145595,"cfile_init":1,"dirs.queue_time_us":9420,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":2182,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3142,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7895,"thread_start_us":9324,"threads_started":1}
I20240426 09:58:08.910950  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(5a863278d07243499fe30d5c68ed0b41): 144674 bytes on disk
I20240426 09:58:08.911732  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(5a863278d07243499fe30d5c68ed0b41) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240426 09:58:08.912521  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014573
W20240426 09:58:09.044974  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240426 09:58:09.322103  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:09.333427  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:58:09.541512  1804 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.455s	user 0.001s	sys 0.000s
I20240426 09:58:09.545517  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.257s	user 0.436s	sys 0.071s Metrics: {"bytes_written":145353,"cfile_init":1,"dirs.queue_time_us":7473,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":8472,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2105,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7872,"thread_start_us":7598,"threads_started":1}
I20240426 09:58:09.546952  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 135281 bytes on disk
I20240426 09:58:09.547601  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240426 09:58:09.548553  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014571
W20240426 09:58:09.649632  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 09:58:10.344702  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240426 09:58:10.671730  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.758s	user 1.507s	sys 0.037s Metrics: {"bytes_written":2511,"cfile_cache_hit":12,"cfile_cache_hit_bytes":47468,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2138276,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":10951,"dirs.run_cpu_time_us":1714,"dirs.run_wall_time_us":16779,"drs_written":1,"lbm_read_time_us":6338,"lbm_reads_lt_1ms":130,"lbm_write_time_us":6360,"lbm_writes_lt_1ms":80,"mutex_wait_us":50,"num_input_rowsets":2,"peak_mem_usage":1056789,"reinsert_count":0,"rows_written":32044,"spinlock_wait_cycles":2816,"thread_start_us":30230,"threads_started":4,"update_count":359}
I20240426 09:58:10.673251  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling MajorDeltaCompactionOp(bbe61252956748109d8ca9a492c04e24): perf score=0.024279
I20240426 09:58:10.712989  2393 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 250 ops behind the committed index  [suppressed 5 similar messages]
I20240426 09:58:10.741047  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: MajorDeltaCompactionOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.068s	user 0.051s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3907,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":78,"lbm_read_time_us":437,"lbm_reads_lt_1ms":24,"lbm_write_time_us":297,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":440}
I20240426 09:58:10.741921  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling MajorDeltaCompactionOp(5a863278d07243499fe30d5c68ed0b41): perf score=0.010482
I20240426 09:58:10.809823  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: MajorDeltaCompactionOp(5a863278d07243499fe30d5c68ed0b41) complete. Timing: real 0.068s	user 0.037s	sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1813,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":72,"lbm_read_time_us":333,"lbm_reads_lt_1ms":20,"lbm_write_time_us":305,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":179}
I20240426 09:58:10.810807  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 595614 bytes on disk
I20240426 09:58:10.812522  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240426 09:58:10.813356  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846): perf score=0.004913
I20240426 09:58:11.013024  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: MajorDeltaCompactionOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.196s	user 0.132s	sys 0.009s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3815,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":90,"lbm_read_time_us":397,"lbm_reads_lt_1ms":20,"lbm_write_time_us":1575,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"update_count":359}
I20240426 09:58:11.014221  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(bbe61252956748109d8ca9a492c04e24): 140583 bytes on disk
I20240426 09:58:11.014863  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(bbe61252956748109d8ca9a492c04e24) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240426 09:58:11.109153  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.560s	user 1.252s	sys 0.032s Metrics: {"bytes_written":590966,"cfile_cache_hit":13,"cfile_cache_hit_bytes":93595,"cfile_cache_miss":93,"cfile_cache_miss_bytes":2102921,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":6329,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1722,"drs_written":1,"lbm_read_time_us":21151,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":123,"lbm_write_time_us":6246,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":1059871,"rows_written":32243,"thread_start_us":6451,"threads_started":1}
I20240426 09:58:11.110244  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 602248 bytes on disk
I20240426 09:58:11.110867  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240426 09:58:11.729332  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240426 09:58:12.000476  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240426 09:58:12.237486  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:12.239363  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:58:12.821507  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240426 09:58:12.959581  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240426 09:58:14.121338  2393 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 422 ops behind the committed index  [suppressed 6 similar messages]
W20240426 09:58:14.182518  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240426 09:58:14.263960  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021058024460140544 Inserted count: 60200
I20240426 09:58:14.264197  1683 linked_list-test-util.h:421] Finished inserting list. Added 60200 in chain
I20240426 09:58:14.264328  1683 linked_list-test-util.h:422] Last entries inserted had keys:
I20240426 09:58:14.264462  1683 linked_list-test-util.h:424] 0: 3083107162024574976
I20240426 09:58:14.264529  1683 linked_list-test-util.h:424] 1: -9136899422574149631
I20240426 09:58:14.264595  1683 linked_list-test-util.h:424] 2: -2909880458486611966
I20240426 09:58:14.264683  1683 linked_list-test-util.h:424] 3: 3317279243089215491
I20240426 09:58:14.264768  1683 linked_list-test-util.h:424] 4: -5819760916973223932
I20240426 09:58:14.264845  1683 linked_list-test-util.h:424] 5: 407398784602603525
I20240426 09:58:14.264923  1683 linked_list-test-util.h:424] 6: 6634417748690141190
I20240426 09:58:14.264981  1683 linked_list-test-util.h:424] 7: -5585588835908583417
I20240426 09:58:14.265049  1683 linked_list-test-util.h:424] 8: 3724537290203529224
I20240426 09:58:14.265103  1683 linked_list-test-util.h:424] 9: -8495187819418484727
I20240426 09:58:14.265174  1683 linked_list-test-util.h:424] 10: -2268028117842657270
I20240426 09:58:14.265236  1683 linked_list-test-util.h:424] 11: 3958709371268169739
I20240426 09:58:14.265306  1683 linked_list-test-util.h:424] 12: -5177908576329269236
I20240426 09:58:14.265362  1683 linked_list-test-util.h:424] 13: 1049110387758268429
I20240426 09:58:14.265501  1683 linked_list-test-util.h:424] 14: 7275847876869095438
I20240426 09:58:14.265563  1683 linked_list-test-util.h:424] 15: -4943736495264628721
I20240426 09:58:14.265631  1683 linked_list-test-util.h:424] 16: 4366248893359194128
I20240426 09:58:14.265686  1683 linked_list-test-util.h:424] 17: -7853616953751240687
I20240426 09:58:14.265754  1683 linked_list-test-util.h:424] 18: -1626597989663703022
I20240426 09:58:14.265808  1683 linked_list-test-util.h:424] 19: 4600561711912124435
I20240426 09:58:14.265892  1683 linked_list-test-util.h:424] 20: -4536196973173604332
I20240426 09:58:14.265949  1683 linked_list-test-util.h:424] 21: 1690540515937222677
I20240426 09:58:14.266016  1683 linked_list-test-util.h:424] 22: 7917700217513050134
I20240426 09:58:14.266077  1683 linked_list-test-util.h:424] 23: -4302024892108963817
I20240426 09:58:14.266144  1683 linked_list-test-util.h:424] 24: 5007819759026438168
I20240426 09:58:14.266197  1683 linked_list-test-util.h:424] 25: -7211905350595575783
I20240426 09:58:14.266274  1683 linked_list-test-util.h:424] 26: -984745649019748326
I20240426 09:58:14.266330  1683 linked_list-test-util.h:424] 27: 8324958264627363867
I20240426 09:58:14.266398  1683 linked_list-test-util.h:424] 28: -3894766844994650084
I20240426 09:58:14.266458  1683 linked_list-test-util.h:424] 29: 2332392856581177373
I20240426 09:58:14.266526  1683 linked_list-test-util.h:424] 30: 8559130345692004382
I20240426 09:58:14.266579  1683 linked_list-test-util.h:424] 31: -577487601905434593
I20240426 09:58:14.266649  1683 linked_list-test-util.h:424] 32: 5649531362182103072
I20240426 09:58:14.266698  1683 linked_list-test-util.h:424] 33: -6570053009951621087
I20240426 09:58:14.266738  1683 linked_list-test-util.h:424] 34: -343315520840794078
I20240426 09:58:14.266770  1683 linked_list-test-util.h:424] 35: 8966669867783028771
I20240426 09:58:14.266819  1683 linked_list-test-util.h:424] 36: -3252914504350695388
I20240426 09:58:14.266862  1683 linked_list-test-util.h:424] 37: 2973822984760131621
I20240426 09:58:14.266920  1683 linked_list-test-util.h:424] 38: 9200982686335959078
I20240426 09:58:14.266964  1683 linked_list-test-util.h:424] 39: 64224001250230311
I20240426 09:58:14.267022  1683 linked_list-test-util.h:424] 40: 6291102227849347112
I20240426 09:58:14.267066  1683 linked_list-test-util.h:424] 41: -5928622881772666839
I20240426 09:58:14.267124  1683 linked_list-test-util.h:424] 42: 298396082314870826
I20240426 09:58:14.267167  1683 linked_list-test-util.h:424] 43: -8838221865282568149
I20240426 09:58:14.267233  1683 linked_list-test-util.h:424] 44: -2611484376171741140
I20240426 09:58:14.267278  1683 linked_list-test-util.h:424] 45: 3615675325404086317
I20240426 09:58:14.267339  1683 linked_list-test-util.h:424] 46: -8604049784217927634
I20240426 09:58:14.267381  1683 linked_list-test-util.h:424] 47: 705794866917474351
I20240426 09:58:14.267438  1683 linked_list-test-util.h:424] 48: 6932813831005012016
I20240426 09:58:14.267480  1683 linked_list-test-util.h:424] 49: -5286911278617001935
I20240426 09:58:14.282930  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:58:14.303634  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125474275551 usec, L: 1 Expected Rows: 18550
I20240426 09:58:14.354115  1683 linked_list-test-util.h:528] Calling callback on tserver 56e7c87ce920461cba4d6cccc793ebd4
I20240426 09:58:14.370337  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling FlushMRSOp(b746c53e16154116b5ee848f02464846): perf score=1.000000
I20240426 09:58:14.645223  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:14.650074  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240426 09:58:14.707804  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240426 09:58:15.017903  1683 meta_cache.cc:1510] marking tablet server 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183) as failed
W20240426 09:58:15.018111  1683 meta_cache.cc:302] tablet b746c53e16154116b5ee848f02464846: replica 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111)
I20240426 09:58:15.156675  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.786s	user 0.424s	sys 0.055s Metrics: {"bytes_written":143024,"cfile_init":1,"dirs.queue_time_us":26255,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1861,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2344,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7786,"thread_start_us":25809,"threads_started":1}
I20240426 09:58:15.157752  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 138139 bytes on disk
I20240426 09:58:15.158421  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240426 09:58:15.159111  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014456
I20240426 09:58:15.248992  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: FlushMRSOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.966s	user 0.453s	sys 0.063s Metrics: {"bytes_written":146072,"cfile_init":1,"dirs.queue_time_us":9467,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":11779,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5702,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7985,"spinlock_wait_cycles":247040,"thread_start_us":9641,"threads_started":1}
I20240426 09:58:15.253100  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 141715 bytes on disk
I20240426 09:58:15.253908  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240426 09:58:15.254904  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling CompactRowSetsOp(b746c53e16154116b5ee848f02464846): perf score=1.014457
I20240426 09:58:15.286598  1683 linked_list-test-util.h:767] Done collecting results (18550 rows in 932.661ms)
I20240426 09:58:15.327330  1683 linked_list-test-util.h:556] Snapshot: P: 1714125474275551 usec, L: 1 verified. Result: OK
I20240426 09:58:15.334187  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125484254611 usec, L: 1 Expected Rows: 39650
W20240426 09:58:15.355456  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240426 09:58:15.454667  1683 linked_list-test-util.h:528] Calling callback on tserver 56e7c87ce920461cba4d6cccc793ebd4
I20240426 09:58:16.390988  1683 linked_list-test-util.h:767] Done collecting results (39650 rows in 936.476ms)
I20240426 09:58:16.487571  1683 linked_list-test-util.h:556] Snapshot: P: 1714125484254611 usec, L: 1 verified. Result: OK
I20240426 09:58:16.491170  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125494252964 usec, L: 1 Expected Rows: 60200
I20240426 09:58:16.501113  1683 linked_list-test-util.h:528] Calling callback on tserver 56e7c87ce920461cba4d6cccc793ebd4
W20240426 09:58:16.609617  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240426 09:58:16.900521  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 1.741s	user 1.588s	sys 0.025s Metrics: {"bytes_written":738761,"cfile_cache_hit":103,"cfile_cache_hit_bytes":2334770,"cfile_cache_miss":15,"cfile_cache_miss_bytes":414460,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1088,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":936,"drs_written":1,"lbm_read_time_us":924,"lbm_reads_lt_1ms":35,"lbm_write_time_us":8661,"lbm_writes_lt_1ms":92,"num_input_rowsets":2,"peak_mem_usage":1330769,"rows_written":40029,"thread_start_us":1266,"threads_started":2,"wal-append.queue_time_us":383}
I20240426 09:58:16.901604  1913 maintenance_manager.cc:419] P 56e7c87ce920461cba4d6cccc793ebd4: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 752387 bytes on disk
I20240426 09:58:16.902237  1804 maintenance_manager.cc:643] P 56e7c87ce920461cba4d6cccc793ebd4: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240426 09:58:17.313462  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: CompactRowSetsOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 2.058s	user 1.745s	sys 0.075s Metrics: {"bytes_written":738761,"cfile_cache_hit":11,"cfile_cache_hit_bytes":1214,"cfile_cache_miss":113,"cfile_cache_miss_bytes":2746966,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":7152,"dirs.run_cpu_time_us":2163,"dirs.run_wall_time_us":18199,"drs_written":1,"lbm_read_time_us":7458,"lbm_reads_lt_1ms":145,"lbm_write_time_us":8492,"lbm_writes_lt_1ms":92,"mutex_wait_us":25,"num_input_rowsets":2,"peak_mem_usage":1330729,"rows_written":40029,"spinlock_wait_cycles":1920,"thread_start_us":18542,"threads_started":5,"wal-append.queue_time_us":796}
I20240426 09:58:17.314862  2044 maintenance_manager.cc:419] P 7a0fdab05e5f434d8c1e538483da228e: Scheduling UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846): 752387 bytes on disk
I20240426 09:58:17.315620  1935 maintenance_manager.cc:643] P 7a0fdab05e5f434d8c1e538483da228e: UndoDeltaBlockGCOp(b746c53e16154116b5ee848f02464846) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240426 09:58:17.398607  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:17.400720  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:17.520076  2433 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 429 ops behind the committed index  [suppressed 6 similar messages]
W20240426 09:58:17.560612  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240426 09:58:17.790990  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 09:58:17.968420  1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240426 09:58:18.220096  1683 linked_list-test-util.h:767] Done collecting results (60200 rows in 1719.09ms)
I20240426 09:58:18.292019  1683 linked_list-test-util.h:556] Snapshot: P: 1714125494252964 usec, L: 1 verified. Result: OK
I20240426 09:58:18.294919  1683 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 60200
W20240426 09:58:19.197132  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240426 09:58:19.324095  2243 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:19.326483  2221 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:19.328374  1683 linked_list-test-util.h:767] Done collecting results (60200 rows in 999.95ms)
I20240426 09:58:19.413796  1683 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240426 09:58:19.414283  1683 linked_list-test-util.h:692] Successfully verified 60200 rows
/home/jenkins-slave/workspace/build_and_test@3/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 57c0b8293f214a6ba5bec482f225d3b3, Rpc address: host: "127.1.164.195" port: 33183 (Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111))
I20240426 09:58:19.431623  1683 webui_checker.cc:88] shutting down CURL thread
W20240426 09:58:19.939992  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240426 09:58:20.555303  1931 consensus_peers.cc:485] T 5a863278d07243499fe30d5c68ed0b41 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 57c0b8293f214a6ba5bec482f225d3b3 (127.1.164.195:33183): Couldn't send request to peer 57c0b8293f214a6ba5bec482f225d3b3. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.195:33183: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240426 09:58:20.720772  2433 consensus_queue.cc:786] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e [LEADER]: Peer 57c0b8293f214a6ba5bec482f225d3b3 is lagging by at least 429 ops behind the committed index  [suppressed 5 similar messages]
I20240426 09:58:20.982959  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1785
I20240426 09:58:20.984316  1907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.266331  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1785
W20240426 09:58:21.326135  1710 connection.cc:535] server connection from 127.1.164.193:45623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240426 09:58:21.326524  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1916
W20240426 09:58:21.326839  1931 consensus_peers.cc:485] T bbe61252956748109d8ca9a492c04e24 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345): Couldn't send request to peer 56e7c87ce920461cba4d6cccc793ebd4. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:38345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 09:58:21.326957  1931 consensus_peers.cc:485] T b746c53e16154116b5ee848f02464846 P 7a0fdab05e5f434d8c1e538483da228e -> Peer 56e7c87ce920461cba4d6cccc793ebd4 (127.1.164.193:38345): Couldn't send request to peer 56e7c87ce920461cba4d6cccc793ebd4. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:38345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:58:21.327505  2038 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.579411  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1916
I20240426 09:58:21.625548  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 1695
I20240426 09:58:21.626680  1756 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:21.748534  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 1695
2024-04-26T09:58:21Z chronyd exiting
I20240426 09:58:21.789584  1683 test_util.cc:175] -----------------------------------------------
I20240426 09:58:21.789702  1683 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714125461789235-1683-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (39933 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240426 09:58:21.791651  1683 test_util.cc:268] Using random seed: -241009271
I20240426 09:58:21.793509  1683 linked_list-test.cc:100] Linked List Test Configuration:
I20240426 09:58:21.793581  1683 linked_list-test.cc:101] --------------
I20240426 09:58:21.793642  1683 linked_list-test.cc:102] 50 chains
I20240426 09:58:21.793692  1683 linked_list-test.cc:103] 3 tablets
I20240426 09:58:21.793756  1683 linked_list-test.cc:104] Mutations on
I20240426 09:58:21.793807  1683 linked_list-test.cc:105] --------------
I20240426 09:58:21.793967  1683 ts_itest-base.cc:112] Starting cluster with:
I20240426 09:58:21.794023  1683 ts_itest-base.cc:113] --------------
I20240426 09:58:21.794088  1683 ts_itest-base.cc:114] 3 tablet servers
I20240426 09:58:21.794137  1683 ts_itest-base.cc:115] 3 replicas per TS
I20240426 09:58:21.794196  1683 ts_itest-base.cc:116] --------------
2024-04-26T09:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-26T09:58:21Z Disabled control of system clock
I20240426 09:58:21.828562  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.164.254:38115
--webserver_interface=127.1.164.254
--webserver_port=0
--builtin_ntp_servers=127.1.164.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.164.254:38115
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240426 09:58:22.189653  2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:58:22.189992  2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:58:22.190157  2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:58:22.197965  2462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240426 09:58:22.198086  2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:58:22.198137  2462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240426 09:58:22.198194  2462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240426 09:58:22.207528  2462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.164.254:38115
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.164.254:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.1.164.254
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:58:22.208892  2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:58:22.210734  2462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:58:22.221663  2468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:22.221690  2467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:22.221632  2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:58:22.222157  2462 server_base.cc:1034] running on GCE node
I20240426 09:58:22.223205  2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:58:22.224659  2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:58:22.225883  2462 hybrid_clock.cc:648] HybridClock initialized: now 1714125502225820 us; error 64 us; skew 500 ppm
I20240426 09:58:22.226210  2462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:58:22.232605  2462 webserver.cc:460] Webserver started at http://127.1.164.254:36853/ using document root <none> and password file <none>
I20240426 09:58:22.233440  2462 fs_manager.cc:359] Metadata directory not provided
I20240426 09:58:22.233572  2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:58:22.233911  2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:58:22.236702  2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/instance:
uuid: "203994aee8664bd696642f54da23141c"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.237510  2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal/instance:
uuid: "203994aee8664bd696642f54da23141c"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.243315  2462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240426 09:58:22.246978  2476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:22.248386  2462 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240426 09:58:22.248570  2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
uuid: "203994aee8664bd696642f54da23141c"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.248780  2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:58:22.277511  2462 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:58:22.278422  2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:58:22.278718  2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:58:22.298297  2462 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.254:38115
I20240426 09:58:22.298327  2527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.254:38115 every 8 connection(s)
I20240426 09:58:22.299746  2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/data/info.pb
I20240426 09:58:22.302503  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 2462
I20240426 09:58:22.302711  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/master-0/wal/instance
I20240426 09:58:22.304136  2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:58:22.317713  2528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c: Bootstrap starting.
I20240426 09:58:22.320830  2528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c: Neither blocks nor log segments found. Creating new log.
I20240426 09:58:22.321872  2528 log.cc:824] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c: Log is configured to *not* fsync() on all Append() calls
I20240426 09:58:22.324455  2528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c: No bootstrap required, opened a new log
I20240426 09:58:22.331030  2528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } }
I20240426 09:58:22.331403  2528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:58:22.331480  2528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203994aee8664bd696642f54da23141c, State: Initialized, Role: FOLLOWER
I20240426 09:58:22.332069  2528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } }
I20240426 09:58:22.332278  2528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:58:22.332378  2528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:58:22.332495  2528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:58:22.334856  2528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } }
I20240426 09:58:22.335335  2528 leader_election.cc:302] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203994aee8664bd696642f54da23141c; no voters: 
I20240426 09:58:22.335884  2528 leader_election.cc:288] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 09:58:22.335994  2533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:58:22.336642  2533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [term 1 LEADER]: Becoming Leader. State: Replica: 203994aee8664bd696642f54da23141c, State: Running, Role: LEADER
I20240426 09:58:22.337249  2533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } }
I20240426 09:58:22.337951  2528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:58:22.340292  2535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203994aee8664bd696642f54da23141c. Latest consensus state: current_term: 1 leader_uuid: "203994aee8664bd696642f54da23141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } } }
I20240426 09:58:22.340665  2535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [sys.catalog]: This master's current role is: LEADER
I20240426 09:58:22.341109  2534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203994aee8664bd696642f54da23141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203994aee8664bd696642f54da23141c" member_type: VOTER last_known_addr { host: "127.1.164.254" port: 38115 } } }
I20240426 09:58:22.341441  2534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c [sys.catalog]: This master's current role is: LEADER
I20240426 09:58:22.344092  2541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:58:22.349284  2541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:58:22.358361  2541 catalog_manager.cc:1347] Generated new cluster ID: 8e6d08eb1047495d9bb53035ad23a0ea
I20240426 09:58:22.358491  2541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:58:22.382509  2541 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:58:22.384354  2541 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:58:22.403095  2541 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 203994aee8664bd696642f54da23141c: Generated new TSK 0
I20240426 09:58:22.404235  2541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:58:22.410081  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.193:0
--local_ip_for_outbound_sockets=127.1.164.193
--webserver_interface=127.1.164.193
--webserver_port=0
--tserver_master_addrs=127.1.164.254:38115
--builtin_ntp_servers=127.1.164.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240426 09:58:22.751138  2552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:58:22.751462  2552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:58:22.751621  2552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:58:22.759507  2552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:58:22.759773  2552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.193
I20240426 09:58:22.769479  2552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.1.164.193
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.1.164.254:38115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.193
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:58:22.770846  2552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:58:22.772593  2552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:58:22.784813  2558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:22.784757  2560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:22.785346  2557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:58:22.785993  2552 server_base.cc:1034] running on GCE node
I20240426 09:58:22.786599  2552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:58:22.787530  2552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:58:22.788758  2552 hybrid_clock.cc:648] HybridClock initialized: now 1714125502788690 us; error 40 us; skew 500 ppm
I20240426 09:58:22.789109  2552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:58:22.791364  2552 webserver.cc:460] Webserver started at http://127.1.164.193:33611/ using document root <none> and password file <none>
I20240426 09:58:22.792150  2552 fs_manager.cc:359] Metadata directory not provided
I20240426 09:58:22.792269  2552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:58:22.792573  2552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:58:22.795226  2552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/instance:
uuid: "dd73384dd6064db8b8fe41483f4f3f70"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.795974  2552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal/instance:
uuid: "dd73384dd6064db8b8fe41483f4f3f70"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.801126  2552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240426 09:58:22.804582  2566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:22.806047  2552 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240426 09:58:22.806196  2552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
uuid: "dd73384dd6064db8b8fe41483f4f3f70"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:22.806375  2552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:58:22.820652  2552 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:58:22.821463  2552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:58:22.821746  2552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:58:22.822701  2552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:58:22.824196  2552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:58:22.824268  2552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:22.824388  2552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:58:22.824450  2552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:22.855846  2552 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.193:35065
I20240426 09:58:22.855885  2678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.193:35065 every 8 connection(s)
I20240426 09:58:22.857244  2552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
I20240426 09:58:22.864998  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 2552
I20240426 09:58:22.865267  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal/instance
I20240426 09:58:22.869144  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.194:0
--local_ip_for_outbound_sockets=127.1.164.194
--webserver_interface=127.1.164.194
--webserver_port=0
--tserver_master_addrs=127.1.164.254:38115
--builtin_ntp_servers=127.1.164.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240426 09:58:22.871415  2679 heartbeater.cc:347] Connected to a master server at 127.1.164.254:38115
I20240426 09:58:22.871747  2679 heartbeater.cc:464] Registering TS with master...
I20240426 09:58:22.872659  2679 heartbeater.cc:511] Master 127.1.164.254:38115 requested a full tablet report, sending...
I20240426 09:58:22.874994  2493 ts_manager.cc:194] Registered new tserver with Master: dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065)
I20240426 09:58:22.876883  2493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.193:40155
W20240426 09:58:23.208487  2683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:58:23.208856  2683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:58:23.209040  2683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:58:23.216890  2683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:58:23.217125  2683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.194
I20240426 09:58:23.226531  2683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.1.164.194
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.1.164.254:38115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.194
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:58:23.227929  2683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:58:23.229687  2683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:58:23.241870  2691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:23.241956  2688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:23.242038  2689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:58:23.242944  2683 server_base.cc:1034] running on GCE node
I20240426 09:58:23.243649  2683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:58:23.244621  2683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:58:23.245838  2683 hybrid_clock.cc:648] HybridClock initialized: now 1714125503245762 us; error 83 us; skew 500 ppm
I20240426 09:58:23.246140  2683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:58:23.248507  2683 webserver.cc:460] Webserver started at http://127.1.164.194:41783/ using document root <none> and password file <none>
I20240426 09:58:23.249294  2683 fs_manager.cc:359] Metadata directory not provided
I20240426 09:58:23.249423  2683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:58:23.249729  2683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:58:23.252435  2683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/instance:
uuid: "969b97eced3443c89bbdbd643283addc"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.253226  2683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal/instance:
uuid: "969b97eced3443c89bbdbd643283addc"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.258672  2683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240426 09:58:23.262274  2697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.263813  2683 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240426 09:58:23.264011  2683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
uuid: "969b97eced3443c89bbdbd643283addc"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.264214  2683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:58:23.278442  2683 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:58:23.279295  2683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:58:23.279593  2683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:58:23.280552  2683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:58:23.282099  2683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:58:23.282179  2683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.282302  2683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:58:23.282356  2683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.323496  2683 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.194:41997
I20240426 09:58:23.323530  2809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.194:41997 every 8 connection(s)
I20240426 09:58:23.325238  2683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/data/info.pb
I20240426 09:58:23.333472  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 2683
I20240426 09:58:23.333653  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-1/wal/instance
I20240426 09:58:23.337348  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.195:0
--local_ip_for_outbound_sockets=127.1.164.195
--webserver_interface=127.1.164.195
--webserver_port=0
--tserver_master_addrs=127.1.164.254:38115
--builtin_ntp_servers=127.1.164.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240426 09:58:23.338711  2810 heartbeater.cc:347] Connected to a master server at 127.1.164.254:38115
I20240426 09:58:23.339071  2810 heartbeater.cc:464] Registering TS with master...
I20240426 09:58:23.339960  2810 heartbeater.cc:511] Master 127.1.164.254:38115 requested a full tablet report, sending...
I20240426 09:58:23.341491  2493 ts_manager.cc:194] Registered new tserver with Master: 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997)
I20240426 09:58:23.342417  2493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.194:46549
W20240426 09:58:23.681267  2813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:58:23.681603  2813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:58:23.681756  2813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:58:23.690083  2813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:58:23.690331  2813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.195
I20240426 09:58:23.699774  2813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.1.164.195
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.1.164.254:38115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.195
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:58:23.701256  2813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:58:23.703102  2813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:58:23.715929  2821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:23.715972  2819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:23.715972  2818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:58:23.716845  2813 server_base.cc:1034] running on GCE node
I20240426 09:58:23.717515  2813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:58:23.718509  2813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:58:23.719750  2813 hybrid_clock.cc:648] HybridClock initialized: now 1714125503719683 us; error 47 us; skew 500 ppm
I20240426 09:58:23.720095  2813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:58:23.722554  2813 webserver.cc:460] Webserver started at http://127.1.164.195:45563/ using document root <none> and password file <none>
I20240426 09:58:23.723318  2813 fs_manager.cc:359] Metadata directory not provided
I20240426 09:58:23.723445  2813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:58:23.723757  2813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:58:23.726508  2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/instance:
uuid: "a8ab44fa842448b4a9aa292ec9b029de"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.727268  2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal/instance:
uuid: "a8ab44fa842448b4a9aa292ec9b029de"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.732694  2813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240426 09:58:23.736282  2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.737771  2813 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240426 09:58:23.737955  2813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
uuid: "a8ab44fa842448b4a9aa292ec9b029de"
format_stamp: "Formatted at 2024-04-26 09:58:23 on dist-test-slave-8sfd"
I20240426 09:58:23.738150  2813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:58:23.757416  2813 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:58:23.758334  2813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:58:23.758652  2813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:58:23.759730  2813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:58:23.761497  2813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:58:23.761608  2813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.761729  2813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:58:23.761775  2813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:23.794998  2813 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.195:45917
I20240426 09:58:23.795056  2939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.195:45917 every 8 connection(s)
I20240426 09:58:23.796367  2813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/data/info.pb
I20240426 09:58:23.801865  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 2813
I20240426 09:58:23.802093  1683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-2/wal/instance
I20240426 09:58:23.810269  2940 heartbeater.cc:347] Connected to a master server at 127.1.164.254:38115
I20240426 09:58:23.810559  2940 heartbeater.cc:464] Registering TS with master...
I20240426 09:58:23.811417  2940 heartbeater.cc:511] Master 127.1.164.254:38115 requested a full tablet report, sending...
I20240426 09:58:23.812768  2493 ts_manager.cc:194] Registered new tserver with Master: a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917)
I20240426 09:58:23.813593  2493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.195:36325
I20240426 09:58:23.818205  1683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:58:23.833994  2493 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43836:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240426 09:58:23.840215  2493 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 09:58:23.872295  2745 tablet_service.cc:1467] Processing CreateTablet for tablet a958a34dd7ea4435bf410b10b4c05267 (DEFAULT_TABLE table=TestTable [id=95ddf8c54e5644a884cbaba870d112eb]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240426 09:58:23.874114  2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a958a34dd7ea4435bf410b10b4c05267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:58:23.877187  2614 tablet_service.cc:1467] Processing CreateTablet for tablet a958a34dd7ea4435bf410b10b4c05267 (DEFAULT_TABLE table=TestTable [id=95ddf8c54e5644a884cbaba870d112eb]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240426 09:58:23.878801  2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a958a34dd7ea4435bf410b10b4c05267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:58:23.880110  2679 heartbeater.cc:503] Master 127.1.164.254:38115 was elected leader, sending a full tablet report...
I20240426 09:58:23.879920  2875 tablet_service.cc:1467] Processing CreateTablet for tablet a958a34dd7ea4435bf410b10b4c05267 (DEFAULT_TABLE table=TestTable [id=95ddf8c54e5644a884cbaba870d112eb]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240426 09:58:23.881253  2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a958a34dd7ea4435bf410b10b4c05267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:58:23.888386  2958 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Bootstrap starting.
I20240426 09:58:23.892169  2959 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Bootstrap starting.
I20240426 09:58:23.892699  2958 tablet_bootstrap.cc:654] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Neither blocks nor log segments found. Creating new log.
I20240426 09:58:23.893853  2958 log.cc:824] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Log is configured to *not* fsync() on all Append() calls
I20240426 09:58:23.895833  2961 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Bootstrap starting.
I20240426 09:58:23.896008  2959 tablet_bootstrap.cc:654] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Neither blocks nor log segments found. Creating new log.
I20240426 09:58:23.897204  2959 log.cc:824] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Log is configured to *not* fsync() on all Append() calls
I20240426 09:58:23.897212  2958 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: No bootstrap required, opened a new log
I20240426 09:58:23.897527  2958 ts_tablet_manager.cc:1367] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240426 09:58:23.899510  2961 tablet_bootstrap.cc:654] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Neither blocks nor log segments found. Creating new log.
I20240426 09:58:23.900748  2961 log.cc:824] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Log is configured to *not* fsync() on all Append() calls
I20240426 09:58:23.901986  2959 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: No bootstrap required, opened a new log
I20240426 09:58:23.902247  2959 ts_tablet_manager.cc:1367] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240426 09:58:23.904579  2961 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: No bootstrap required, opened a new log
I20240426 09:58:23.904892  2961 ts_tablet_manager.cc:1367] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240426 09:58:23.906443  2958 raft_consensus.cc:348] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.906894  2958 raft_consensus.cc:374] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:58:23.907047  2958 raft_consensus.cc:729] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969b97eced3443c89bbdbd643283addc, State: Initialized, Role: FOLLOWER
I20240426 09:58:23.907755  2958 consensus_queue.cc:260] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.908994  2810 heartbeater.cc:503] Master 127.1.164.254:38115 was elected leader, sending a full tablet report...
I20240426 09:58:23.908993  2959 raft_consensus.cc:348] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.909346  2959 raft_consensus.cc:374] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:58:23.909443  2959 raft_consensus.cc:729] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73384dd6064db8b8fe41483f4f3f70, State: Initialized, Role: FOLLOWER
I20240426 09:58:23.909564  2958 ts_tablet_manager.cc:1398] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.005s
I20240426 09:58:23.910171  2959 consensus_queue.cc:260] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.914165  2959 ts_tablet_manager.cc:1398] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240426 09:58:23.917820  2961 raft_consensus.cc:348] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.918292  2961 raft_consensus.cc:374] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:58:23.918406  2961 raft_consensus.cc:729] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ab44fa842448b4a9aa292ec9b029de, State: Initialized, Role: FOLLOWER
I20240426 09:58:23.919006  2961 consensus_queue.cc:260] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.920050  2940 heartbeater.cc:503] Master 127.1.164.254:38115 was elected leader, sending a full tablet report...
I20240426 09:58:23.920432  2961 ts_tablet_manager.cc:1398] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.004s
I20240426 09:58:23.937283  2966 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 09:58:23.937577  2966 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.939520  2966 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065)
I20240426 09:58:23.945988  2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a958a34dd7ea4435bf410b10b4c05267" candidate_uuid: "a8ab44fa842448b4a9aa292ec9b029de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969b97eced3443c89bbdbd643283addc" is_pre_election: true
I20240426 09:58:23.946556  2765 raft_consensus.cc:2454] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8ab44fa842448b4a9aa292ec9b029de in term 0.
I20240426 09:58:23.946803  2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a958a34dd7ea4435bf410b10b4c05267" candidate_uuid: "a8ab44fa842448b4a9aa292ec9b029de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73384dd6064db8b8fe41483f4f3f70" is_pre_election: true
I20240426 09:58:23.947243  2828 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de; no voters: 
I20240426 09:58:23.947316  2634 raft_consensus.cc:2454] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8ab44fa842448b4a9aa292ec9b029de in term 0.
I20240426 09:58:23.947611  2966 raft_consensus.cc:2781] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 09:58:23.947742  2966 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 09:58:23.947850  2966 raft_consensus.cc:3037] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:58:23.950423  2966 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.951088  2966 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [CANDIDATE]: Term 1 election: Requested vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065)
I20240426 09:58:23.951551  2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a958a34dd7ea4435bf410b10b4c05267" candidate_uuid: "a8ab44fa842448b4a9aa292ec9b029de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969b97eced3443c89bbdbd643283addc"
I20240426 09:58:23.951740  2765 raft_consensus.cc:3037] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:58:23.951809  2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a958a34dd7ea4435bf410b10b4c05267" candidate_uuid: "a8ab44fa842448b4a9aa292ec9b029de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73384dd6064db8b8fe41483f4f3f70"
I20240426 09:58:23.952055  2634 raft_consensus.cc:3037] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:58:23.954404  2765 raft_consensus.cc:2454] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ab44fa842448b4a9aa292ec9b029de in term 1.
I20240426 09:58:23.954603  2634 raft_consensus.cc:2454] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ab44fa842448b4a9aa292ec9b029de in term 1.
I20240426 09:58:23.954891  2828 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de; no voters: 
I20240426 09:58:23.955228  2966 raft_consensus.cc:2781] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:58:23.955667  2966 raft_consensus.cc:686] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [term 1 LEADER]: Becoming Leader. State: Replica: a8ab44fa842448b4a9aa292ec9b029de, State: Running, Role: LEADER
I20240426 09:58:23.956210  2966 consensus_queue.cc:237] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:23.960966  2493 catalog_manager.cc:5484] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de reported cstate change: term changed from 0 to 1, leader changed from <none> to a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195). New cstate: current_term: 1 leader_uuid: "a8ab44fa842448b4a9aa292ec9b029de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } health_report { overall_health: UNKNOWN } } }
I20240426 09:58:23.979985  1683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240426 09:58:23.982189  1683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dd73384dd6064db8b8fe41483f4f3f70 to finish bootstrapping
I20240426 09:58:23.991067  1683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 969b97eced3443c89bbdbd643283addc to finish bootstrapping
I20240426 09:58:23.999244  1683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a8ab44fa842448b4a9aa292ec9b029de to finish bootstrapping
I20240426 09:58:24.006968  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 2552
I20240426 09:58:24.007951  2674 generic_service.cc:196] Checking for leaks (request via RPC)
W20240426 09:58:24.048152  2941 tablet.cc:2369] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240426 09:58:24.077487  2811 tablet.cc:2369] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:58:24.095782  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 2552
I20240426 09:58:24.116479  1683 linked_list-test-util.h:401] Running for: 30.000s
I20240426 09:58:24.116621  1683 linked_list-test-util.h:402] Sampling every 10000000 us
I20240426 09:58:24.116720  1683 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
W20240426 09:58:24.140848  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111)
I20240426 09:58:24.141598  2765 raft_consensus.cc:1261] T a958a34dd7ea4435bf410b10b4c05267 P 969b97eced3443c89bbdbd643283addc [term 1 FOLLOWER]: Refusing update from remote peer a8ab44fa842448b4a9aa292ec9b029de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 09:58:24.142457  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 09:58:24.142582  2968 consensus_queue.cc:1035] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Connected to new peer: Peer: permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 09:58:26.327595  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 09:58:28.743970  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 09:58:29.258694  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240426 09:58:29.793839  2966 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 50 ops behind the committed index 
W20240426 09:58:31.355455  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 09:58:32.966445  2968 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 365 ops behind the committed index  [suppressed 5 similar messages]
W20240426 09:58:33.961668  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 09:58:34.121279  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021058105817874432 Inserted count: 48450
W20240426 09:58:34.561493  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240426 09:58:36.342780  2968 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 675 ops behind the committed index  [suppressed 6 similar messages]
W20240426 09:58:36.351234  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 09:58:39.028227  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240426 09:58:39.459946  3052 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 973 ops behind the committed index  [suppressed 5 similar messages]
W20240426 09:58:39.987794  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240426 09:58:41.766872  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240426 09:58:42.878227  2980 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 1301 ops behind the committed index  [suppressed 5 similar messages]
I20240426 09:58:44.117290  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021058146762334208 Inserted count: 95250
W20240426 09:58:44.442453  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240426 09:58:45.057245  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20240426 09:58:46.129483  2966 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 1606 ops behind the committed index  [suppressed 5 similar messages]
W20240426 09:58:47.171836  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240426 09:58:49.524693  3110 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 1935 ops behind the committed index  [suppressed 6 similar messages]
W20240426 09:58:49.532893  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240426 09:58:50.557427  2829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240426 09:58:52.076077  2829 consensus_peers.cc:485] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de -> Peer dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065): Couldn't send request to peer dd73384dd6064db8b8fe41483f4f3f70. Status: Network error: Client connection negotiation failed: client connection to 127.1.164.193:35065: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240426 09:58:52.660979  3129 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 2236 ops behind the committed index  [suppressed 5 similar messages]
I20240426 09:58:54.119062  1683 linked_list-test-util.h:417] Sample at HT timestamp: 7021058187728822272 Inserted count: 142200
I20240426 09:58:54.119213  1683 linked_list-test-util.h:421] Finished inserting list. Added 142200 in chain
I20240426 09:58:54.119292  1683 linked_list-test-util.h:422] Last entries inserted had keys:
I20240426 09:58:54.119374  1683 linked_list-test-util.h:424] 0: -6718955018436018176
I20240426 09:58:54.119433  1683 linked_list-test-util.h:424] 1: 2231290202049085441
I20240426 09:58:54.119503  1683 linked_list-test-util.h:424] 2: -7265349388663783422
I20240426 09:58:54.119585  1683 linked_list-test-util.h:424] 3: 1684755094332899331
I20240426 09:58:54.119653  1683 linked_list-test-util.h:424] 4: 3915904558893694980
I20240426 09:58:54.119709  1683 linked_list-test-util.h:424] 5: -5580735031819173883
I20240426 09:58:54.119776  1683 linked_list-test-util.h:424] 6: 3369650926154219526
I20240426 09:58:54.119834  1683 linked_list-test-util.h:424] 7: -6127129402046939129
I20240426 09:58:54.119894  1683 linked_list-test-util.h:424] 8: -3895839199997853688
I20240426 09:58:54.119943  1683 linked_list-test-util.h:424] 9: 5054265282998829065
I20240426 09:58:54.120014  1683 linked_list-test-util.h:424] 10: -4442515045202329590
I20240426 09:58:54.120127  1683 linked_list-test-util.h:424] 11: 4507870912771063819
I20240426 09:58:54.120254  1683 linked_list-test-util.h:424] 12: 6739161114820149260
I20240426 09:58:54.120309  1683 linked_list-test-util.h:424] 13: -2757478475892719603
I20240426 09:58:54.120393  1683 linked_list-test-util.h:424] 14: 6192485269615673358
I20240426 09:58:54.120445  1683 linked_list-test-util.h:424] 15: -3304154321097195505
I20240426 09:58:54.120514  1683 linked_list-test-util.h:424] 16: -1072864119048110064
I20240426 09:58:54.120573  1683 linked_list-test-util.h:424] 17: 7877381101436993553
I20240426 09:58:54.120647  1683 linked_list-test-util.h:424] 18: -1619258489275875310
I20240426 09:58:54.120697  1683 linked_list-test-util.h:424] 19: 7330845993720807443
I20240426 09:58:54.120860  1683 linked_list-test-util.h:424] 20: -8884748615427948524
I20240426 09:58:54.120920  1683 linked_list-test-util.h:424] 21: 65355867568734229
I20240426 09:58:54.120991  1683 linked_list-test-util.h:424] 22: 9015741825542127638
I20240426 09:58:54.121042  1683 linked_list-test-util.h:424] 23: -481038502659031017
I20240426 09:58:54.121106  1683 linked_list-test-util.h:424] 24: 1750251699390054424
I20240426 09:58:54.121156  1683 linked_list-test-util.h:424] 25: -7746387891322814439
I20240426 09:58:54.121229  1683 linked_list-test-util.h:424] 26: 1203716591673868314
I20240426 09:58:54.121285  1683 linked_list-test-util.h:424] 27: 3434866056234663963
I20240426 09:58:54.121356  1683 linked_list-test-util.h:424] 28: -6061492059501494244
I20240426 09:58:54.121404  1683 linked_list-test-util.h:424] 29: 2888612423495188509
I20240426 09:58:54.121471  1683 linked_list-test-util.h:424] 30: -6608167904705970146
I20240426 09:58:54.121521  1683 linked_list-test-util.h:424] 31: -4376877702656884705
I20240426 09:58:54.121584  1683 linked_list-test-util.h:424] 32: 4573226780339798048
I20240426 09:58:54.121634  1683 linked_list-test-util.h:424] 33: -4923272072884649951
I20240426 09:58:54.121696  1683 linked_list-test-util.h:424] 34: 4026832410112032802
I20240426 09:58:54.121745  1683 linked_list-test-util.h:424] 35: 6258122612161118243
I20240426 09:58:54.121809  1683 linked_list-test-util.h:424] 36: -3238516978551750620
I20240426 09:58:54.121867  1683 linked_list-test-util.h:424] 37: 5711446766956642341
I20240426 09:58:54.121948  1683 linked_list-test-util.h:424] 38: -3784911348779515866
I20240426 09:58:54.122004  1683 linked_list-test-util.h:424] 39: -1553621146730430425
I20240426 09:58:54.122061  1683 linked_list-test-util.h:424] 40: 7396342598777962536
I20240426 09:58:54.122104  1683 linked_list-test-util.h:424] 41: -2100296991934906327
I20240426 09:58:54.122148  1683 linked_list-test-util.h:424] 42: 6849807491061776426
I20240426 09:58:54.122181  1683 linked_list-test-util.h:424] 43: 9080956955622572075
I20240426 09:58:54.122224  1683 linked_list-test-util.h:424] 44: -415401160113586132
I20240426 09:58:54.122260  1683 linked_list-test-util.h:424] 45: 8534703322883096621
I20240426 09:58:54.122315  1683 linked_list-test-util.h:424] 46: -962077005318062034
I20240426 09:58:54.122367  1683 linked_list-test-util.h:424] 47: 1269213196731023407
I20240426 09:58:54.122416  1683 linked_list-test-util.h:424] 48: -8227426393981845456
I20240426 09:58:54.122448  1683 linked_list-test-util.h:424] 49: 722959563991547953
I20240426 09:58:54.161063  1683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
/tmp/dist-test-taskzQdrQa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.164.193:35065
--local_ip_for_outbound_sockets=127.1.164.193
--tserver_master_addrs=127.1.164.254:38115
--webserver_port=33611
--webserver_interface=127.1.164.193
--builtin_ntp_servers=127.1.164.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240426 09:58:54.519224  3160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240426 09:58:54.519528  3160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240426 09:58:54.519644  3160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240426 09:58:54.527192  3160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240426 09:58:54.527421  3160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.164.193
I20240426 09:58:54.538105  3160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.164.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.164.193:35065
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.1.164.193
--webserver_port=33611
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.1.164.254:38115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.164.193
--log_dir=/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 058c4bd848277b49fc34b19c81c39a215d7f637f
build type FASTDEBUG
built by None at 26 Apr 2024 09:35:23 UTC on efd1337a2aa3
build id 258
ASAN enabled
I20240426 09:58:54.539517  3160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:58:54.541275  3160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:58:54.553812  3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:54.553923  3165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:58:54.553961  3168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:58:54.554522  3160 server_base.cc:1034] running on GCE node
I20240426 09:58:54.555186  3160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240426 09:58:54.556161  3160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240426 09:58:54.557384  3160 hybrid_clock.cc:648] HybridClock initialized: now 1714125534557306 us; error 52 us; skew 500 ppm
I20240426 09:58:54.557680  3160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:58:54.565956  3160 webserver.cc:460] Webserver started at http://127.1.164.193:33611/ using document root <none> and password file <none>
I20240426 09:58:54.566730  3160 fs_manager.cc:359] Metadata directory not provided
I20240426 09:58:54.566879  3160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:58:54.572526  3160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.003s
I20240426 09:58:54.575542  3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:58:54.576975  3160 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240426 09:58:54.577136  3160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
uuid: "dd73384dd6064db8b8fe41483f4f3f70"
format_stamp: "Formatted at 2024-04-26 09:58:22 on dist-test-slave-8sfd"
I20240426 09:58:54.578019  3160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:58:54.607710  3160 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:58:54.608610  3160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:58:54.608932  3160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:58:54.609916  3160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:58:54.611799  3181 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240426 09:58:54.617265  3160 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240426 09:58:54.617401  3160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240426 09:58:54.617540  3160 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240426 09:58:54.620612  3160 ts_tablet_manager.cc:590] Registered 1 tablets
I20240426 09:58:54.620699  3160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
I20240426 09:58:54.621228  3181 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Bootstrap starting.
I20240426 09:58:54.640128  3181 log.cc:824] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Log is configured to *not* fsync() on all Append() calls
I20240426 09:58:54.643306  3181 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 09:58:54.643743  3181 log.cc:1225] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/wal/wals/a958a34dd7ea4435bf410b10b4c05267.recovery-1714125534643529
I20240426 09:58:54.643855  3181 tablet_bootstrap.cc:492] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Bootstrap complete.
I20240426 09:58:54.644321  3181 ts_tablet_manager.cc:1367] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.014s
I20240426 09:58:54.654460  3181 raft_consensus.cc:348] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:54.655381  3181 raft_consensus.cc:729] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73384dd6064db8b8fe41483f4f3f70, State: Initialized, Role: FOLLOWER
I20240426 09:58:54.656147  3181 consensus_queue.cc:260] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:54.657806  3181 ts_tablet_manager.cc:1398] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Time spent starting tablet: real 0.013s	user 0.012s	sys 0.001s
I20240426 09:58:54.668542  3160 rpc_server.cc:307] RPC server started. Bound to: 127.1.164.193:35065
I20240426 09:58:54.668613  3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.164.193:35065 every 8 connection(s)
I20240426 09:58:54.669981  3160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzQdrQa/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714125461789235-1683-0/linked-list-cluster/ts-0/data/info.pb
W20240426 09:58:54.671264  3291 tablet.cc:2369] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 09:58:54.677944  1683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu as pid 3160
I20240426 09:58:54.683095  3290 heartbeater.cc:347] Connected to a master server at 127.1.164.254:38115
I20240426 09:58:54.683531  3290 heartbeater.cc:464] Registering TS with master...
I20240426 09:58:54.684608  3290 heartbeater.cc:511] Master 127.1.164.254:38115 requested a full tablet report, sending...
I20240426 09:58:54.687713  2493 ts_manager.cc:194] Re-registered known tserver with Master: dd73384dd6064db8b8fe41483f4f3f70 (127.1.164.193:35065)
I20240426 09:58:54.688977  2493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.164.193:37057
I20240426 09:58:54.708258  1683 cluster_itest_util.cc:256] Not converged past 2844 yet: 0.0 1.2875 1.2875
I20240426 09:58:54.812093  1683 cluster_itest_util.cc:256] Not converged past 2844 yet: 0.0 1.2875 1.2875
I20240426 09:58:54.844681  3290 heartbeater.cc:503] Master 127.1.164.254:38115 was elected leader, sending a full tablet report...
I20240426 09:58:55.024657  1683 cluster_itest_util.cc:256] Not converged past 2844 yet: 1.503 1.2875 1.2875
I20240426 09:58:55.454046  1683 cluster_itest_util.cc:256] Not converged past 2844 yet: 1.1494 1.2875 1.2875
I20240426 09:58:55.815821  3110 consensus_queue.cc:786] T a958a34dd7ea4435bf410b10b4c05267 P a8ab44fa842448b4a9aa292ec9b029de [LEADER]: Peer dd73384dd6064db8b8fe41483f4f3f70 is lagging by at least 417 ops behind the committed index  [suppressed 7 similar messages]
I20240426 09:58:55.845609  3299 mvcc.cc:205] Tried to move back new op lower bound from 7021058086087553024 to 7021058064209121280. Current Snapshot: MvccSnapshot[applied={T|T < 7021058086087553024}]
I20240426 09:58:56.029569  1683 cluster_itest_util.cc:256] Not converged past 2844 yet: 1.2458 1.2875 1.2875
I20240426 09:58:56.533038  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 2683
I20240426 09:58:56.534060  2805 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:56.893214  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 2683
W20240426 09:58:56.978346  2828 connection.cc:535] client connection to 127.1.164.194:41997 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240426 09:58:56.978547  2828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 7 similar messages]
I20240426 09:58:56.985641  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 2813
I20240426 09:58:56.987007  2935 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:58:57.324343  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 2813
W20240426 09:58:57.385291  3177 connection.cc:535] server connection from 127.1.164.195:33777 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240426 09:58:57.390427  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125514115692 usec, L: 1 Expected Rows: 48450
I20240426 09:58:57.482545  1683 linked_list-test-util.h:528] Calling callback on tserver dd73384dd6064db8b8fe41483f4f3f70
I20240426 09:58:58.839062  3316 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8ab44fa842448b4a9aa292ec9b029de)
I20240426 09:58:58.839462  3316 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:58:58.842481  3316 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917)
W20240426 09:58:58.843458  3175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111)
W20240426 09:58:58.854213  3175 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997): Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111)
W20240426 09:58:58.856500  3177 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917): Network error: Client connection negotiation failed: client connection to 127.1.164.195:45917: connect: Connection refused (error 111)
I20240426 09:58:58.856662  3177 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd73384dd6064db8b8fe41483f4f3f70; no voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de
I20240426 09:58:58.857069  3316 raft_consensus.cc:2726] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:58:59.499866  1683 linked_list-test-util.h:767] Done collecting results (48450 rows in 2017.47ms)
I20240426 09:58:59.593588  1683 linked_list-test-util.h:556] Snapshot: P: 1714125514115692 usec, L: 1 verified. Result: OK
I20240426 09:58:59.597188  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125524111898 usec, L: 1 Expected Rows: 95250
I20240426 09:58:59.607770  1683 linked_list-test-util.h:528] Calling callback on tserver dd73384dd6064db8b8fe41483f4f3f70
I20240426 09:59:00.370090  3320 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8ab44fa842448b4a9aa292ec9b029de)
I20240426 09:59:00.370290  3320 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:59:00.371315  3320 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917)
W20240426 09:59:00.373644  3177 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917): Network error: Client connection negotiation failed: client connection to 127.1.164.195:45917: connect: Connection refused (error 111)
W20240426 09:59:00.373927  3175 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997): Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111)
I20240426 09:59:00.374017  3175 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd73384dd6064db8b8fe41483f4f3f70; no voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de
I20240426 09:59:00.374277  3320 raft_consensus.cc:2726] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:59:01.667302  1683 linked_list-test-util.h:767] Done collecting results (95250 rows in 2059.68ms)
I20240426 09:59:01.783008  1683 linked_list-test-util.h:556] Snapshot: P: 1714125524111898 usec, L: 1 verified. Result: OK
I20240426 09:59:01.785979  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 1714125534113482 usec, L: 1 Expected Rows: 142200
I20240426 09:59:01.793565  1683 linked_list-test-util.h:528] Calling callback on tserver dd73384dd6064db8b8fe41483f4f3f70
I20240426 09:59:02.437834  3324 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8ab44fa842448b4a9aa292ec9b029de)
I20240426 09:59:02.438271  3324 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:59:02.438918  3324 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917)
W20240426 09:59:02.440884  3175 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997): Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111)
W20240426 09:59:02.441020  3177 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917): Network error: Client connection negotiation failed: client connection to 127.1.164.195:45917: connect: Connection refused (error 111)
I20240426 09:59:02.441123  3177 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd73384dd6064db8b8fe41483f4f3f70; no voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de
I20240426 09:59:02.441392  3324 raft_consensus.cc:2726] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:59:03.918951  1683 linked_list-test-util.h:767] Done collecting results (142200 rows in 2125.52ms)
I20240426 09:59:04.091495  1683 linked_list-test-util.h:556] Snapshot: P: 1714125534113482 usec, L: 1 verified. Result: OK
I20240426 09:59:04.094866  1683 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 142200
I20240426 09:59:04.098059  1683 linked_list-test-util.h:528] Calling callback on tserver dd73384dd6064db8b8fe41483f4f3f70
I20240426 09:59:04.882062  3328 raft_consensus.cc:482] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8ab44fa842448b4a9aa292ec9b029de)
I20240426 09:59:04.882256  3328 raft_consensus.cc:504] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969b97eced3443c89bbdbd643283addc" member_type: VOTER last_known_addr { host: "127.1.164.194" port: 41997 } } peers { permanent_uuid: "a8ab44fa842448b4a9aa292ec9b029de" member_type: VOTER last_known_addr { host: "127.1.164.195" port: 45917 } } peers { permanent_uuid: "dd73384dd6064db8b8fe41483f4f3f70" member_type: VOTER last_known_addr { host: "127.1.164.193" port: 35065 } }
I20240426 09:59:04.882961  3328 leader_election.cc:288] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997), a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917)
W20240426 09:59:04.883769  3175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240426 09:59:04.885499  3175 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969b97eced3443c89bbdbd643283addc (127.1.164.194:41997): Network error: Client connection negotiation failed: client connection to 127.1.164.194:41997: connect: Connection refused (error 111)
W20240426 09:59:04.885648  3177 leader_election.cc:334] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8ab44fa842448b4a9aa292ec9b029de (127.1.164.195:45917): Network error: Client connection negotiation failed: client connection to 127.1.164.195:45917: connect: Connection refused (error 111)
I20240426 09:59:04.885792  3177 leader_election.cc:302] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd73384dd6064db8b8fe41483f4f3f70; no voters: 969b97eced3443c89bbdbd643283addc, a8ab44fa842448b4a9aa292ec9b029de
I20240426 09:59:04.886019  3328 raft_consensus.cc:2726] T a958a34dd7ea4435bf410b10b4c05267 P dd73384dd6064db8b8fe41483f4f3f70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 09:59:06.140477  1683 linked_list-test-util.h:767] Done collecting results (142200 rows in 2042.55ms)
I20240426 09:59:06.312659  1683 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240426 09:59:06.313541  1683 linked_list-test-util.h:692] Successfully verified 142200 rows
I20240426 09:59:06.314617  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 3160
I20240426 09:59:06.326537  3285 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:59:06.563444  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 3160
I20240426 09:59:06.615132  1683 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu pid 2462
I20240426 09:59:06.616357  2523 generic_service.cc:196] Checking for leaks (request via RPC)
I20240426 09:59:06.726101  1683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzQdrQa/build/asan/bin/kudu with pid 2462
2024-04-26T09:59:06Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (44982 ms)
[----------] 2 tests from LinkedListTest (84915 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (84915 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST
I20240426 09:59:06.773237  1683 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~51 seconds ago