[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:11:00.027787 26073 test_util.cc:268] Using random seed: 115464524
I20240501 08:11:00.040205 26073 linked_list-test.cc:100] Linked List Test Configuration:
I20240501 08:11:00.040308 26073 linked_list-test.cc:101] --------------
I20240501 08:11:00.040349 26073 linked_list-test.cc:102] 50 chains
I20240501 08:11:00.040414 26073 linked_list-test.cc:103] 3 tablets
I20240501 08:11:00.040478 26073 linked_list-test.cc:104] Mutations on
I20240501 08:11:00.040529 26073 linked_list-test.cc:105] --------------
I20240501 08:11:00.040781 26073 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:00.040848 26073 ts_itest-base.cc:113] --------------
I20240501 08:11:00.040906 26073 ts_itest-base.cc:114] 3 tablet servers
I20240501 08:11:00.040956 26073 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:00.041015 26073 ts_itest-base.cc:116] --------------
2024-05-01T08:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:00Z Disabled control of system clock
I20240501 08:11:00.065268 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.118.126:43115
--webserver_interface=127.25.118.126
--webserver_port=0
--builtin_ntp_servers=127.25.118.84:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.118.126:43115
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240501 08:11:00.415525 26085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:00.415856 26085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:00.415959 26085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:00.424531 26085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:00.424679 26085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:00.424728 26085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:00.424785 26085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:00.435011 26085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.118.126:43115
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.118.126:43115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.25.118.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:00.436442 26085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:00.438236 26085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:00.450096 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.450698 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.451187 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:00.451876 26085 server_base.cc:1034] running on GCE node
I20240501 08:11:00.452715 26085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:00.454186 26085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:00.455410 26085 hybrid_clock.cc:648] HybridClock initialized: now 1714551060455346 us; error 43 us; skew 500 ppm
I20240501 08:11:00.455739 26085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:00.464577 26085 webserver.cc:460] Webserver started at http://127.25.118.126:44773/ using document root <none> and password file <none>
I20240501 08:11:00.465391 26085 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:00.465530 26085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:00.465824 26085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:00.468688 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/instance:
uuid: "511c4023558f4c9898db563978d07e99"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-2ds8"
I20240501 08:11:00.469440 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal/instance:
uuid: "511c4023558f4c9898db563978d07e99"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-2ds8"
I20240501 08:11:00.475081 26085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 08:11:00.478893 26099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.480510 26085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:00.480698 26085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
uuid: "511c4023558f4c9898db563978d07e99"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-2ds8"
I20240501 08:11:00.480903 26085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:00.514366 26085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:00.515357 26085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:00.515686 26085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:00.535624 26085 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.126:43115
I20240501 08:11:00.535640 26150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.126:43115 every 8 connection(s)
I20240501 08:11:00.537214 26085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
I20240501 08:11:00.542016 26151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:00.543547 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 26085
I20240501 08:11:00.543789 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal/instance
I20240501 08:11:00.553366 26151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99: Bootstrap starting.
I20240501 08:11:00.556519 26151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:00.557561 26151 log.cc:824] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:00.560490 26151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99: No bootstrap required, opened a new log
I20240501 08:11:00.567575 26151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } }
I20240501 08:11:00.568099 26151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:00.568205 26151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511c4023558f4c9898db563978d07e99, State: Initialized, Role: FOLLOWER
I20240501 08:11:00.568981 26151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } }
I20240501 08:11:00.569267 26151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:00.569413 26151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:00.569587 26151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:00.572343 26151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } }
I20240501 08:11:00.572991 26151 leader_election.cc:302] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 511c4023558f4c9898db563978d07e99; no voters:
I20240501 08:11:00.573676 26151 leader_election.cc:288] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:00.573889 26156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:00.576004 26156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [term 1 LEADER]: Becoming Leader. State: Replica: 511c4023558f4c9898db563978d07e99, State: Running, Role: LEADER
I20240501 08:11:00.576524 26151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:00.576541 26156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } }
I20240501 08:11:00.579653 26158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 511c4023558f4c9898db563978d07e99. Latest consensus state: current_term: 1 leader_uuid: "511c4023558f4c9898db563978d07e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } } }
I20240501 08:11:00.579715 26157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "511c4023558f4c9898db563978d07e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511c4023558f4c9898db563978d07e99" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 43115 } } }
I20240501 08:11:00.580019 26158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:00.580068 26157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:00.586398 26164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:00.592273 26164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:00.603463 26164 catalog_manager.cc:1347] Generated new cluster ID: ec6917c0d47d48238f96c93a93e022af
I20240501 08:11:00.603591 26164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:00.628791 26164 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:00.630219 26164 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:00.640899 26164 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 511c4023558f4c9898db563978d07e99: Generated new TSK 0
I20240501 08:11:00.641880 26164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:00.653122 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.65:0
--local_ip_for_outbound_sockets=127.25.118.65
--webserver_interface=127.25.118.65
--webserver_port=0
--tserver_master_addrs=127.25.118.126:43115
--builtin_ntp_servers=127.25.118.84:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240501 08:11:01.015003 26175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:01.015312 26175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:01.015463 26175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:01.023586 26175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:01.023751 26175 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 08:11:01.023818 26175 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 08:11:01.023902 26175 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 08:11:01.024017 26175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.65
I20240501 08:11:01.034226 26175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.25.118.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.25.118.126:43115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.65
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:01.035830 26175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:01.037744 26175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:01.050494 26180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:01.050819 26183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:01.051465 26181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:01.051932 26175 server_base.cc:1034] running on GCE node
I20240501 08:11:01.052548 26175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:01.053550 26175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:01.054769 26175 hybrid_clock.cc:648] HybridClock initialized: now 1714551061054718 us; error 46 us; skew 500 ppm
I20240501 08:11:01.055122 26175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:01.057529 26175 webserver.cc:460] Webserver started at http://127.25.118.65:35329/ using document root <none> and password file <none>
I20240501 08:11:01.058286 26175 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:01.058388 26175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:01.058739 26175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:01.061466 26175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/instance:
uuid: "ef205d178a5e41058e69ab8cc77cb795"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.062203 26175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/instance:
uuid: "ef205d178a5e41058e69ab8cc77cb795"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.067656 26175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 08:11:01.071184 26189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.072692 26175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:11:01.072854 26175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
uuid: "ef205d178a5e41058e69ab8cc77cb795"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.073037 26175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:01.104689 26175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:01.105562 26175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:01.105829 26175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:01.106797 26175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:01.108345 26175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:01.108429 26175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.108517 26175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:01.108569 26175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.143450 26175 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.65:40247
I20240501 08:11:01.143527 26301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.65:40247 every 8 connection(s)
I20240501 08:11:01.145328 26175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
I20240501 08:11:01.152042 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 26175
I20240501 08:11:01.152292 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/instance
I20240501 08:11:01.156219 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.66:0
--local_ip_for_outbound_sockets=127.25.118.66
--webserver_interface=127.25.118.66
--webserver_port=0
--tserver_master_addrs=127.25.118.126:43115
--builtin_ntp_servers=127.25.118.84:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 08:11:01.160094 26302 heartbeater.cc:347] Connected to a master server at 127.25.118.126:43115
I20240501 08:11:01.160425 26302 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:01.161294 26302 heartbeater.cc:511] Master 127.25.118.126:43115 requested a full tablet report, sending...
I20240501 08:11:01.163723 26116 ts_manager.cc:194] Registered new tserver with Master: ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247)
I20240501 08:11:01.166460 26116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.65:33161
W20240501 08:11:01.511541 26306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:01.511868 26306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:01.511989 26306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:01.520076 26306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:01.520246 26306 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 08:11:01.520294 26306 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 08:11:01.520354 26306 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 08:11:01.520434 26306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.66
I20240501 08:11:01.530239 26306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.25.118.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.25.118.126:43115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.66
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:01.531723 26306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:01.533516 26306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:01.547740 26311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:01.547960 26312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:01.548884 26314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:01.553305 26306 server_base.cc:1034] running on GCE node
I20240501 08:11:01.554001 26306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:01.555116 26306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:01.556385 26306 hybrid_clock.cc:648] HybridClock initialized: now 1714551061556323 us; error 51 us; skew 500 ppm
I20240501 08:11:01.556732 26306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:01.559311 26306 webserver.cc:460] Webserver started at http://127.25.118.66:45729/ using document root <none> and password file <none>
I20240501 08:11:01.560106 26306 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:01.560215 26306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:01.560530 26306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:01.563410 26306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/instance:
uuid: "a60f7f39696f4125b1fe90f5ac68cae1"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.564237 26306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal/instance:
uuid: "a60f7f39696f4125b1fe90f5ac68cae1"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.569890 26306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:01.573733 26320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.575863 26306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:01.576045 26306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
uuid: "a60f7f39696f4125b1fe90f5ac68cae1"
format_stamp: "Formatted at 2024-05-01 08:11:01 on dist-test-slave-2ds8"
I20240501 08:11:01.576308 26306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:01.593560 26306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:01.594468 26306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:01.594805 26306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:01.595808 26306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:01.597376 26306 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:01.597473 26306 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.597579 26306 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:01.598130 26306 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:01.631668 26306 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.66:35327
I20240501 08:11:01.631714 26432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.66:35327 every 8 connection(s)
I20240501 08:11:01.633044 26306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
I20240501 08:11:01.636065 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 26306
I20240501 08:11:01.636257 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal/instance
I20240501 08:11:01.639351 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.67:0
--local_ip_for_outbound_sockets=127.25.118.67
--webserver_interface=127.25.118.67
--webserver_port=0
--tserver_master_addrs=127.25.118.126:43115
--builtin_ntp_servers=127.25.118.84:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240501 08:11:01.647481 26433 heartbeater.cc:347] Connected to a master server at 127.25.118.126:43115
I20240501 08:11:01.647809 26433 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:01.648640 26433 heartbeater.cc:511] Master 127.25.118.126:43115 requested a full tablet report, sending...
I20240501 08:11:01.650139 26116 ts_manager.cc:194] Registered new tserver with Master: a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327)
I20240501 08:11:01.651243 26116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.66:54139
W20240501 08:11:01.993237 26436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:01.993541 26436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:01.993659 26436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:02.001560 26436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:02.001724 26436 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240501 08:11:02.001775 26436 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240501 08:11:02.001833 26436 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240501 08:11:02.001912 26436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.67
I20240501 08:11:02.011677 26436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.25.118.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.25.118.126:43115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.67
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:02.013002 26436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:02.014794 26436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:02.027271 26442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:02.027482 26441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:02.028034 26444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:02.028599 26436 server_base.cc:1034] running on GCE node
I20240501 08:11:02.029281 26436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:02.030318 26436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:02.031533 26436 hybrid_clock.cc:648] HybridClock initialized: now 1714551062031467 us; error 67 us; skew 500 ppm
I20240501 08:11:02.031860 26436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:02.034243 26436 webserver.cc:460] Webserver started at http://127.25.118.67:42211/ using document root <none> and password file <none>
I20240501 08:11:02.035131 26436 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:02.035254 26436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:02.035554 26436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:02.042889 26436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/instance:
uuid: "ce059656f8cb4229845d41328f2957d9"
format_stamp: "Formatted at 2024-05-01 08:11:02 on dist-test-slave-2ds8"
I20240501 08:11:02.043694 26436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal/instance:
uuid: "ce059656f8cb4229845d41328f2957d9"
format_stamp: "Formatted at 2024-05-01 08:11:02 on dist-test-slave-2ds8"
I20240501 08:11:02.049319 26436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 08:11:02.053170 26450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:02.054708 26436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:02.055371 26436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
uuid: "ce059656f8cb4229845d41328f2957d9"
format_stamp: "Formatted at 2024-05-01 08:11:02 on dist-test-slave-2ds8"
I20240501 08:11:02.055588 26436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:02.095520 26436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:02.096417 26436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:02.096709 26436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:02.097735 26436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:02.099398 26436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:02.099503 26436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:02.099609 26436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:02.099676 26436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:02.134951 26436 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.67:45999
I20240501 08:11:02.135011 26562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.67:45999 every 8 connection(s)
I20240501 08:11:02.136348 26436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
I20240501 08:11:02.142714 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 26436
I20240501 08:11:02.142961 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal/instance
I20240501 08:11:02.150507 26563 heartbeater.cc:347] Connected to a master server at 127.25.118.126:43115
I20240501 08:11:02.150837 26563 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:02.151679 26563 heartbeater.cc:511] Master 127.25.118.126:43115 requested a full tablet report, sending...
I20240501 08:11:02.153142 26116 ts_manager.cc:194] Registered new tserver with Master: ce059656f8cb4229845d41328f2957d9 (127.25.118.67:45999)
I20240501 08:11:02.153968 26116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.67:55039
I20240501 08:11:02.160598 26073 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 08:11:02.169894 26302 heartbeater.cc:503] Master 127.25.118.126:43115 was elected leader, sending a full tablet report...
I20240501 08:11:02.184422 26116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36286:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240501 08:11:02.191728 26116 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:11:02.226572 26498 tablet_service.cc:1467] Processing CreateTablet for tablet 6665039606404fe0a8e6c45674aadca4 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 08:11:02.226572 26497 tablet_service.cc:1467] Processing CreateTablet for tablet eef0447a12844339b7e999cb340d96e6 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 08:11:02.226773 26496 tablet_service.cc:1467] Processing CreateTablet for tablet 7f42407c614045e7bc0dbef2b85e2511 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 08:11:02.229187 26497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef0447a12844339b7e999cb340d96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.230036 26498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665039606404fe0a8e6c45674aadca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.232868 26496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f42407c614045e7bc0dbef2b85e2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.231966 26367 tablet_service.cc:1467] Processing CreateTablet for tablet eef0447a12844339b7e999cb340d96e6 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 08:11:02.231814 26368 tablet_service.cc:1467] Processing CreateTablet for tablet 6665039606404fe0a8e6c45674aadca4 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 08:11:02.233992 26367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef0447a12844339b7e999cb340d96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.231662 26366 tablet_service.cc:1467] Processing CreateTablet for tablet 7f42407c614045e7bc0dbef2b85e2511 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 08:11:02.235278 26366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f42407c614045e7bc0dbef2b85e2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.235560 26368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665039606404fe0a8e6c45674aadca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.238907 26235 tablet_service.cc:1467] Processing CreateTablet for tablet 7f42407c614045e7bc0dbef2b85e2511 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240501 08:11:02.240777 26235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f42407c614045e7bc0dbef2b85e2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.242905 26237 tablet_service.cc:1467] Processing CreateTablet for tablet 6665039606404fe0a8e6c45674aadca4 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240501 08:11:02.243522 26237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665039606404fe0a8e6c45674aadca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.246780 26236 tablet_service.cc:1467] Processing CreateTablet for tablet eef0447a12844339b7e999cb340d96e6 (DEFAULT_TABLE table=TestTable [id=54e23bd0b6e04d56ae4a6b39b7d6d06d]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240501 08:11:02.247291 26236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef0447a12844339b7e999cb340d96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.253597 26582 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Bootstrap starting.
I20240501 08:11:02.257612 26582 tablet_bootstrap.cc:654] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.259069 26582 log.cc:824] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:02.266891 26584 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: Bootstrap starting.
I20240501 08:11:02.266909 26581 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: Bootstrap starting.
I20240501 08:11:02.267944 26582 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: No bootstrap required, opened a new log
I20240501 08:11:02.268229 26582 ts_tablet_manager.cc:1367] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240501 08:11:02.270661 26584 tablet_bootstrap.cc:654] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.271210 26581 tablet_bootstrap.cc:654] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.272179 26584 log.cc:824] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:02.272917 26581 log.cc:824] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:02.275470 26581 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: No bootstrap required, opened a new log
I20240501 08:11:02.275578 26584 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: No bootstrap required, opened a new log
I20240501 08:11:02.275768 26581 ts_tablet_manager.cc:1367] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240501 08:11:02.275924 26584 ts_tablet_manager.cc:1367] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240501 08:11:02.275751 26582 raft_consensus.cc:348] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.276243 26582 raft_consensus.cc:374] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.276360 26582 raft_consensus.cc:729] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60f7f39696f4125b1fe90f5ac68cae1, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.277134 26582 consensus_queue.cc:260] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.278381 26433 heartbeater.cc:503] Master 127.25.118.126:43115 was elected leader, sending a full tablet report...
I20240501 08:11:02.281275 26582 ts_tablet_manager.cc:1398] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:11:02.281819 26582 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1: Bootstrap starting.
I20240501 08:11:02.283735 26582 tablet_bootstrap.cc:654] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.283738 26581 raft_consensus.cc:348] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.284297 26581 raft_consensus.cc:374] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.284407 26581 raft_consensus.cc:729] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce059656f8cb4229845d41328f2957d9, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.285110 26581 consensus_queue.cc:260] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.285010 26584 raft_consensus.cc:348] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.285481 26584 raft_consensus.cc:374] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.285686 26584 raft_consensus.cc:729] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef205d178a5e41058e69ab8cc77cb795, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.286759 26584 consensus_queue.cc:260] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.291375 26563 heartbeater.cc:503] Master 127.25.118.126:43115 was elected leader, sending a full tablet report...
I20240501 08:11:02.292074 26581 ts_tablet_manager.cc:1398] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9: Time spent starting tablet: real 0.016s user 0.006s sys 0.011s
I20240501 08:11:02.292095 26584 ts_tablet_manager.cc:1398] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795: Time spent starting tablet: real 0.016s user 0.009s sys 0.003s
I20240501 08:11:02.292552 26581 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Bootstrap starting.
I20240501 08:11:02.293493 26584 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795: Bootstrap starting.
I20240501 08:11:02.294356 26581 tablet_bootstrap.cc:654] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.295367 26584 tablet_bootstrap.cc:654] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.298451 26582 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1: No bootstrap required, opened a new log
I20240501 08:11:02.298668 26582 ts_tablet_manager.cc:1367] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20240501 08:11:02.299810 26582 raft_consensus.cc:348] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.300076 26582 raft_consensus.cc:374] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.300169 26582 raft_consensus.cc:729] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60f7f39696f4125b1fe90f5ac68cae1, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.300436 26582 consensus_queue.cc:260] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.301692 26582 ts_tablet_manager.cc:1398] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:02.302565 26581 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: No bootstrap required, opened a new log
I20240501 08:11:02.302772 26581 ts_tablet_manager.cc:1367] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240501 08:11:02.303432 26582 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1: Bootstrap starting.
I20240501 08:11:02.303701 26581 raft_consensus.cc:348] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.303972 26581 raft_consensus.cc:374] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.304085 26581 raft_consensus.cc:729] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce059656f8cb4229845d41328f2957d9, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.304293 26584 tablet_bootstrap.cc:492] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795: No bootstrap required, opened a new log
I20240501 08:11:02.304381 26581 consensus_queue.cc:260] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.304518 26584 ts_tablet_manager.cc:1367] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 08:11:02.304939 26581 ts_tablet_manager.cc:1398] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:02.305179 26582 tablet_bootstrap.cc:654] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.305641 26584 raft_consensus.cc:348] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.305881 26581 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9: Bootstrap starting.
I20240501 08:11:02.305892 26584 raft_consensus.cc:374] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.306016 26584 raft_consensus.cc:729] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef205d178a5e41058e69ab8cc77cb795, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.306370 26584 consensus_queue.cc:260] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.307579 26581 tablet_bootstrap.cc:654] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.307915 26584 ts_tablet_manager.cc:1398] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:02.308266 26584 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Bootstrap starting.
I20240501 08:11:02.311553 26584 tablet_bootstrap.cc:654] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.312819 26581 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9: No bootstrap required, opened a new log
I20240501 08:11:02.313040 26581 ts_tablet_manager.cc:1367] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:11:02.313834 26581 raft_consensus.cc:348] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.314107 26581 raft_consensus.cc:374] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.314216 26581 raft_consensus.cc:729] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce059656f8cb4229845d41328f2957d9, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.314215 26582 tablet_bootstrap.cc:492] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1: No bootstrap required, opened a new log
I20240501 08:11:02.314402 26582 ts_tablet_manager.cc:1367] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 08:11:02.314493 26581 consensus_queue.cc:260] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.315191 26581 ts_tablet_manager.cc:1398] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:02.315246 26582 raft_consensus.cc:348] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.315549 26582 raft_consensus.cc:374] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.315639 26582 raft_consensus.cc:729] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60f7f39696f4125b1fe90f5ac68cae1, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.315909 26582 consensus_queue.cc:260] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.316614 26582 ts_tablet_manager.cc:1398] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:02.318048 26584 tablet_bootstrap.cc:492] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: No bootstrap required, opened a new log
I20240501 08:11:02.318222 26584 ts_tablet_manager.cc:1367] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 08:11:02.318825 26584 raft_consensus.cc:348] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.319032 26584 raft_consensus.cc:374] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.319124 26584 raft_consensus.cc:729] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef205d178a5e41058e69ab8cc77cb795, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.319391 26584 consensus_queue.cc:260] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.320003 26584 ts_tablet_manager.cc:1398] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:02.321661 26588 raft_consensus.cc:482] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:02.321890 26588 raft_consensus.cc:504] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.323621 26588 leader_election.cc:288] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247), a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327)
I20240501 08:11:02.331205 26257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef0447a12844339b7e999cb340d96e6" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795" is_pre_election: true
I20240501 08:11:02.331816 26257 raft_consensus.cc:2454] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 0.
I20240501 08:11:02.332127 26388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef0447a12844339b7e999cb340d96e6" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" is_pre_election: true
I20240501 08:11:02.332469 26453 leader_election.cc:302] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce059656f8cb4229845d41328f2957d9, ef205d178a5e41058e69ab8cc77cb795; no voters:
I20240501 08:11:02.332631 26388 raft_consensus.cc:2454] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 0.
I20240501 08:11:02.332885 26588 raft_consensus.cc:2781] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:02.333050 26588 raft_consensus.cc:482] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:02.333156 26588 raft_consensus.cc:3037] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.336050 26588 raft_consensus.cc:504] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.336656 26588 leader_election.cc:288] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 election: Requested vote from peers ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247), a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327)
I20240501 08:11:02.337227 26257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef0447a12844339b7e999cb340d96e6" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795"
I20240501 08:11:02.337256 26388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef0447a12844339b7e999cb340d96e6" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60f7f39696f4125b1fe90f5ac68cae1"
I20240501 08:11:02.337425 26257 raft_consensus.cc:3037] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.337427 26388 raft_consensus.cc:3037] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.339634 26388 raft_consensus.cc:2454] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 1.
I20240501 08:11:02.339637 26257 raft_consensus.cc:2454] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 1.
I20240501 08:11:02.340088 26453 leader_election.cc:302] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60f7f39696f4125b1fe90f5ac68cae1, ce059656f8cb4229845d41328f2957d9; no voters:
I20240501 08:11:02.340262 26587 raft_consensus.cc:482] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:02.340407 26588 raft_consensus.cc:2781] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.340521 26587 raft_consensus.cc:504] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.340914 26588 raft_consensus.cc:686] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [term 1 LEADER]: Becoming Leader. State: Replica: ce059656f8cb4229845d41328f2957d9, State: Running, Role: LEADER
I20240501 08:11:02.341063 26598 raft_consensus.cc:482] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:02.341279 26598 raft_consensus.cc:504] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.341562 26588 consensus_queue.cc:237] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.342128 26598 leader_election.cc:288] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327), ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247)
I20240501 08:11:02.342537 26587 leader_election.cc:288] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247), ce059656f8cb4229845d41328f2957d9 (127.25.118.67:45999)
I20240501 08:11:02.354605 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce059656f8cb4229845d41328f2957d9" is_pre_election: true
I20240501 08:11:02.355125 26518 raft_consensus.cc:2454] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60f7f39696f4125b1fe90f5ac68cae1 in term 0.
I20240501 08:11:02.355705 26324 leader_election.cc:302] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60f7f39696f4125b1fe90f5ac68cae1, ce059656f8cb4229845d41328f2957d9; no voters:
I20240501 08:11:02.355935 26257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665039606404fe0a8e6c45674aadca4" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795" is_pre_election: true
I20240501 08:11:02.356154 26587 raft_consensus.cc:2781] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:02.356168 26257 raft_consensus.cc:2454] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 0.
I20240501 08:11:02.356268 26587 raft_consensus.cc:482] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:02.356348 26587 raft_consensus.cc:3037] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.356643 26453 leader_election.cc:302] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce059656f8cb4229845d41328f2957d9, ef205d178a5e41058e69ab8cc77cb795; no voters:
I20240501 08:11:02.356170 26114 catalog_manager.cc:5484] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce059656f8cb4229845d41328f2957d9 (127.25.118.67). New cstate: current_term: 1 leader_uuid: "ce059656f8cb4229845d41328f2957d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:02.357965 26388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665039606404fe0a8e6c45674aadca4" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" is_pre_election: true
I20240501 08:11:02.358095 26588 raft_consensus.cc:2781] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:02.358211 26588 raft_consensus.cc:482] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:02.358330 26588 raft_consensus.cc:3037] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.358701 26388 raft_consensus.cc:2454] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 0.
I20240501 08:11:02.360586 26587 raft_consensus.cc:504] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.361732 26588 raft_consensus.cc:504] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.361963 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce059656f8cb4229845d41328f2957d9"
I20240501 08:11:02.362133 26518 raft_consensus.cc:3037] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.362403 26257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795" is_pre_election: true
I20240501 08:11:02.362468 26256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795"
I20240501 08:11:02.362656 26257 raft_consensus.cc:2454] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60f7f39696f4125b1fe90f5ac68cae1 in term 0.
I20240501 08:11:02.363566 26588 leader_election.cc:288] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 election: Requested vote from peers a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327), ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247)
I20240501 08:11:02.363982 26256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665039606404fe0a8e6c45674aadca4" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795"
I20240501 08:11:02.364182 26256 raft_consensus.cc:3037] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.364140 26388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665039606404fe0a8e6c45674aadca4" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60f7f39696f4125b1fe90f5ac68cae1"
I20240501 08:11:02.364354 26388 raft_consensus.cc:3037] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.364897 26518 raft_consensus.cc:2454] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60f7f39696f4125b1fe90f5ac68cae1 in term 1.
I20240501 08:11:02.365012 26587 leader_election.cc:288] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [CANDIDATE]: Term 1 election: Requested vote from peers ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247), ce059656f8cb4229845d41328f2957d9 (127.25.118.67:45999)
I20240501 08:11:02.365410 26324 leader_election.cc:302] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a60f7f39696f4125b1fe90f5ac68cae1, ce059656f8cb4229845d41328f2957d9; no voters: ef205d178a5e41058e69ab8cc77cb795
I20240501 08:11:02.365753 26587 raft_consensus.cc:2781] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.366974 26587 raft_consensus.cc:686] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 LEADER]: Becoming Leader. State: Replica: a60f7f39696f4125b1fe90f5ac68cae1, State: Running, Role: LEADER
I20240501 08:11:02.367026 26256 raft_consensus.cc:2454] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 1.
I20240501 08:11:02.367048 26388 raft_consensus.cc:2454] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce059656f8cb4229845d41328f2957d9 in term 1.
I20240501 08:11:02.367518 26453 leader_election.cc:302] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60f7f39696f4125b1fe90f5ac68cae1, ce059656f8cb4229845d41328f2957d9; no voters:
I20240501 08:11:02.367584 26587 consensus_queue.cc:237] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:02.367799 26588 raft_consensus.cc:2781] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.367941 26588 raft_consensus.cc:686] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [term 1 LEADER]: Becoming Leader. State: Replica: ce059656f8cb4229845d41328f2957d9, State: Running, Role: LEADER
I20240501 08:11:02.368208 26588 consensus_queue.cc:237] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } }
I20240501 08:11:02.370772 26114 catalog_manager.cc:5484] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce059656f8cb4229845d41328f2957d9 (127.25.118.67). New cstate: current_term: 1 leader_uuid: "ce059656f8cb4229845d41328f2957d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:02.372750 26116 catalog_manager.cc:5484] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66). New cstate: current_term: 1 leader_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:02.385100 26434 tablet.cc:2369] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:02.387989 26564 tablet.cc:2369] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:02.398272 26303 tablet.cc:2369] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:02.405068 26073 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 08:11:02.408243 26073 ts_itest-base.cc:233] Waiting for 3 tablets on tserver ef205d178a5e41058e69ab8cc77cb795 to finish bootstrapping
I20240501 08:11:02.418984 26073 ts_itest-base.cc:233] Waiting for 3 tablets on tserver a60f7f39696f4125b1fe90f5ac68cae1 to finish bootstrapping
I20240501 08:11:02.427063 26073 ts_itest-base.cc:233] Waiting for 3 tablets on tserver ce059656f8cb4229845d41328f2957d9 to finish bootstrapping
I20240501 08:11:02.435998 26606 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.25.118.66:45729/maintenance-manager
http://127.25.118.65:35329/startup
http://127.25.118.65:35329/stacks
http://127.25.118.66:45729/mem-trackers
http://127.25.118.126:44773/ipki-ca-cert-der
http://127.25.118.126:44773/ipki-ca-cert-pem
http://127.25.118.65:35329/threadz?group=all
http://127.25.118.66:45729/transactions?tablet_id=7f42407c614045e7bc0dbef2b85e2511
http://127.25.118.67:42211/mem-trackers
http://127.25.118.66:45729/threadz
http://127.25.118.65:35329/mem-trackers
http://127.25.118.67:42211/metrics
http://127.25.118.126:44773/startup
http://127.25.118.65:35329/tablets
http://127.25.118.65:35329/scans
http://127.25.118.66:45729/threadz?group=all
http://127.25.118.126:44773/ipki-ca-cert
http://127.25.118.126:44773/threadz?group=all
http://127.25.118.65:35329/metrics
http://127.25.118.65:35329/maintenance-manager
http://127.25.118.126:44773/dump-entities
http://127.25.118.67:42211/startup
http://127.25.118.65:35329/threadz
http://127.25.118.65:35329/transactions?tablet_id=7f42407c614045e7bc0dbef2b85e2511
http://127.25.118.67:42211/tablets
http://127.25.118.67:42211/transactions?tablet_id=7f42407c614045e7bc0dbef2b85e2511
http://127.25.118.126:44773/threadz
http://127.25.118.126:44773/metrics
http://127.25.118.126:44773/stacks
http://127.25.118.67:42211/maintenance-manager
http://127.25.118.66:45729/stacks
http://127.25.118.126:44773/tablet-servers
http://127.25.118.67:42211/stacks
http://127.25.118.66:45729/startup
http://127.25.118.66:45729/scans
http://127.25.118.66:45729/tablets
http://127.25.118.126:44773/mem-trackers
http://127.25.118.126:44773/masters
http://127.25.118.126:44773/tables
http://127.25.118.67:42211/threadz?group=all
http://127.25.118.67:42211/threadz
http://127.25.118.67:42211/scans
http://127.25.118.66:45729/metrics
I20240501 08:11:02.441365 26073 linked_list-test-util.h:401] Running for: 30.000s
I20240501 08:11:02.441540 26073 linked_list-test-util.h:402] Sampling every 10000000 us
I20240501 08:11:02.441720 26073 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240501 08:11:02.479265 26256 raft_consensus.cc:1261] T eef0447a12844339b7e999cb340d96e6 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Refusing update from remote peer ce059656f8cb4229845d41328f2957d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.479321 26518 raft_consensus.cc:1261] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Refusing update from remote peer a60f7f39696f4125b1fe90f5ac68cae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.480057 26587 consensus_queue.cc:1035] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.480203 26256 raft_consensus.cc:1261] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Refusing update from remote peer ce059656f8cb4229845d41328f2957d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.480309 26588 consensus_queue.cc:1035] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.480777 26598 consensus_queue.cc:1035] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.482105 26388 raft_consensus.cc:1261] T eef0447a12844339b7e999cb340d96e6 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Refusing update from remote peer ce059656f8cb4229845d41328f2957d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.483323 26387 raft_consensus.cc:1261] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1 [term 1 FOLLOWER]: Refusing update from remote peer ce059656f8cb4229845d41328f2957d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.484676 26598 consensus_queue.cc:1035] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.485186 26598 consensus_queue.cc:1035] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.486862 26256 raft_consensus.cc:3037] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.499037 26256 raft_consensus.cc:1261] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Refusing update from remote peer a60f7f39696f4125b1fe90f5ac68cae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:02.499861 26587 consensus_queue.cc:1035] T 7f42407c614045e7bc0dbef2b85e2511 P a60f7f39696f4125b1fe90f5ac68cae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.511384 26613 mvcc.cc:205] Tried to move back new op lower bound from 7022801151905071104 to 7022801151462211584. Current Snapshot: MvccSnapshot[applied={T|T < 7022801151905071104}]
I20240501 08:11:02.515470 26612 mvcc.cc:205] Tried to move back new op lower bound from 7022801151905071104 to 7022801151462211584. Current Snapshot: MvccSnapshot[applied={T|T < 7022801151905071104}]
I20240501 08:11:02.530032 26617 mvcc.cc:205] Tried to move back new op lower bound from 7022801151905071104 to 7022801151462211584. Current Snapshot: MvccSnapshot[applied={T|T < 7022801151905071104}]
I20240501 08:11:03.223027 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:05.404043 26607 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:05.461108 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:06.342391 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:07.900838 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:07.900884 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:07.912837 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:09.386494 26607 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:09.467216 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:10.209926 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 2.308s user 0.563s sys 0.093s Metrics: {"bytes_written":1892,"cfile_init":1,"compiler_manager_pool.queue_time_us":11434,"delete_count":0,"dirs.queue_time_us":5110,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":3837,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10052,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8131,"thread_start_us":23891,"threads_started":2,"update_count":268}
I20240501 08:11:10.212069 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 142831 bytes on disk
I20240501 08:11:10.215003 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 08:11:10.215924 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.014841
I20240501 08:11:10.394253 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 2.493s user 0.548s sys 0.092s Metrics: {"bytes_written":1679,"cfile_init":1,"compiler_manager_pool.queue_time_us":333153,"compiler_manager_pool.run_cpu_time_us":666242,"compiler_manager_pool.run_wall_time_us":2306058,"delete_count":0,"dirs.queue_time_us":27220,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":4179,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":34415,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8098,"thread_start_us":48685,"threads_started":2,"update_count":235}
I20240501 08:11:10.396380 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.013229
I20240501 08:11:10.437288 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.215s user 0.057s sys 0.029s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2540,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":7538,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":1766,"lbm_read_time_us":399,"lbm_reads_lt_1ms":20,"lbm_write_time_us":2734,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"thread_start_us":23659,"threads_started":2,"update_count":268}
I20240501 08:11:10.516250 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 2.603s user 0.569s sys 0.086s Metrics: {"bytes_written":1892,"cfile_init":1,"compiler_manager_pool.queue_time_us":20303,"delete_count":0,"dirs.queue_time_us":9582,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":7824,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3211,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8131,"thread_start_us":41668,"threads_started":2,"update_count":268}
I20240501 08:11:10.518409 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 142831 bytes on disk
I20240501 08:11:10.522213 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 08:11:10.523983 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.014841
I20240501 08:11:10.526129 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.129s user 0.059s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2260,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":472,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1010,"lbm_read_time_us":460,"lbm_reads_lt_1ms":24,"lbm_write_time_us":278,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":3546,"threads_started":1,"update_count":235}
I20240501 08:11:10.529274 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 142506 bytes on disk
I20240501 08:11:10.530150 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 08:11:10.760543 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.236s user 0.062s sys 0.028s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2540,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":23142,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1137,"lbm_read_time_us":425,"lbm_reads_lt_1ms":20,"lbm_write_time_us":328,"lbm_writes_lt_1ms":8,"mutex_wait_us":43,"reinsert_count":0,"spinlock_wait_cycles":37632,"thread_start_us":23194,"threads_started":2,"update_count":268}
I20240501 08:11:11.349555 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:12.458880 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801192736886784 Inserted count: 20300
I20240501 08:11:13.734743 26607 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:13.780972 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:14.197965 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:14.271000 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:14.288975 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:14.947579 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.749s user 0.462s sys 0.096s Metrics: {"bytes_written":146036,"cfile_init":1,"dirs.queue_time_us":6944,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":2290,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1954,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7926,"thread_start_us":7105,"threads_started":1}
I20240501 08:11:14.959985 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014817
I20240501 08:11:15.500984 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.212s user 0.474s sys 0.124s Metrics: {"bytes_written":3101,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":44856,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1733,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2909,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8087,"spinlock_wait_cycles":10112,"thread_start_us":44743,"threads_started":1,"update_count":454}
I20240501 08:11:15.502749 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014816
I20240501 08:11:15.607860 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:16.012362 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.741s user 0.469s sys 0.100s Metrics: {"bytes_written":2900,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":25681,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":2346,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2396,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8024,"thread_start_us":25580,"threads_started":1,"update_count":424}
I20240501 08:11:16.014065 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 139887 bytes on disk
I20240501 08:11:16.020862 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 08:11:16.021821 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014816
I20240501 08:11:16.122452 26705 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60480 (ReqId={client: 906a460dc09f4e52a65c41794a46a25e, seq_no=1535, attempt_no=0}) took 1175 ms. Trace:
I20240501 08:11:16.123931 26705 rpcz_store.cc:276] 0501 08:11:14.946877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:14.947194 (+ 317us) service_pool.cc:225] Handling call
0501 08:11:16.122426 (+1175232us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 08:11:14.950050 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6665039606404fe0a8e6c45674aadca4
0501 08:11:14.950194 (+ 144us) write_op.cc:424] Acquiring schema lock in shared mode
0501 08:11:14.950201 (+ 7us) write_op.cc:427] Acquired schema lock
0501 08:11:14.950204 (+ 3us) tablet.cc:660] Decoding operations
0501 08:11:15.013389 (+ 63185us) write_op.cc:612] Acquiring the partition lock for write op
0501 08:11:15.013407 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0501 08:11:15.013413 (+ 6us) tablet.cc:683] Acquiring locks for 1180 operations
0501 08:11:15.033161 (+ 19748us) tablet.cc:699] Row locks acquired
0501 08:11:15.033166 (+ 5us) write_op.cc:260] PREPARE: finished
0501 08:11:15.033252 (+ 86us) write_op.cc:270] Start()
0501 08:11:15.033300 (+ 48us) write_op.cc:276] Timestamp: P: 1714551075033223 usec, L: 0
0501 08:11:15.033304 (+ 4us) op_driver.cc:331] REPLICATION: starting
0501 08:11:15.033606 (+ 302us) log.cc:842] Serialized 13325 byte log entry
0501 08:11:15.043944 (+ 10338us) op_driver.cc:444] REPLICATION: finished
0501 08:11:15.044117 (+ 173us) write_op.cc:301] APPLY: starting
0501 08:11:15.044138 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0501 08:11:15.502735 (+458597us) tablet.cc:1368] finished BulkCheckPresence
0501 08:11:15.502763 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 08:11:16.101806 (+599043us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 08:11:16.102649 (+ 843us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=2361,key_file_lookups=653,delta_file_lookups=0,mrs_lookups=854
0501 08:11:16.102659 (+ 10us) write_op.cc:312] APPLY: finished
0501 08:11:16.104345 (+ 1686us) log.cc:842] Serialized 7752 byte log entry
0501 08:11:16.104463 (+ 118us) write_op.cc:481] Releasing partition, row and schema locks
0501 08:11:16.122237 (+ 17774us) write_op.cc:446] Released schema lock
0501 08:11:16.122316 (+ 79us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":120,"cfile_cache_hit":1599,"cfile_cache_hit_bytes":31596345,"cfile_cache_miss":2,"cfile_cache_miss_bytes":46601,"lbm_read_time_us":117,"lbm_reads_lt_1ms":2,"num_ops":1180,"prepare.queue_time_us":2285,"prepare.run_cpu_time_us":16160,"prepare.run_wall_time_us":124934,"raft.queue_time_us":128,"raft.run_cpu_time_us":832,"raft.run_wall_time_us":834,"replication_time_us":10605,"spinlock_wait_cycles":1792}]]}
I20240501 08:11:16.543790 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.584s user 0.649s sys 0.032s Metrics: {"bytes_written":2284,"cfile_cache_hit":14,"cfile_cache_hit_bytes":142823,"cfile_cache_miss":60,"cfile_cache_miss_bytes":926753,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":43850,"dirs.run_cpu_time_us":823,"dirs.run_wall_time_us":11984,"drs_written":1,"lbm_read_time_us":2907,"lbm_reads_lt_1ms":96,"lbm_write_time_us":3734,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":521884,"reinsert_count":0,"rows_written":16057,"thread_start_us":46518,"threads_started":4,"update_count":326}
I20240501 08:11:16.545274 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 293313 bytes on disk
I20240501 08:11:16.547225 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 08:11:16.548244 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.009102
I20240501 08:11:16.714447 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.166s user 0.085s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3284,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":8777,"lbm_read_time_us":391,"lbm_reads_lt_1ms":20,"lbm_write_time_us":331,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":326}
I20240501 08:11:16.999567 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.976s user 0.568s sys 0.066s Metrics: {"bytes_written":292535,"cfile_cache_hit":14,"cfile_cache_hit_bytes":96278,"cfile_cache_miss":64,"cfile_cache_miss_bytes":984384,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":11008,"dirs.run_cpu_time_us":1027,"dirs.run_wall_time_us":11583,"drs_written":1,"lbm_read_time_us":3032,"lbm_reads_lt_1ms":100,"lbm_write_time_us":3575,"lbm_writes_lt_1ms":51,"mutex_wait_us":250,"num_input_rowsets":2,"peak_mem_usage":531006,"rows_written":16155,"spinlock_wait_cycles":13696,"thread_start_us":49392,"threads_started":3}
I20240501 08:11:17.000532 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 298459 bytes on disk
I20240501 08:11:17.001164 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240501 08:11:17.059082 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.555s user 0.637s sys 0.027s Metrics: {"bytes_written":293063,"cfile_cache_hit":14,"cfile_cache_hit_bytes":95802,"cfile_cache_miss":62,"cfile_cache_miss_bytes":987072,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":8100,"dirs.run_cpu_time_us":870,"dirs.run_wall_time_us":8102,"drs_written":1,"lbm_read_time_us":2968,"lbm_reads_lt_1ms":102,"lbm_write_time_us":17308,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":532004,"rows_written":16185,"thread_start_us":8812,"threads_started":5}
I20240501 08:11:17.060199 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 299033 bytes on disk
I20240501 08:11:17.060931 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240501 08:11:18.630852 26607 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:18.686975 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:19.729565 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:20.476691 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:20.762826 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:20.822093 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:21.066972 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.590s user 0.387s sys 0.057s Metrics: {"bytes_written":145469,"cfile_init":1,"dirs.queue_time_us":18195,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":21694,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1906,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7652,"spinlock_wait_cycles":1920,"thread_start_us":17771,"threads_started":1}
I20240501 08:11:21.068744 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014708
I20240501 08:11:21.773434 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.010s user 0.441s sys 0.030s Metrics: {"bytes_written":3847,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15472,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":6167,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2705,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7893,"thread_start_us":15499,"threads_started":1,"update_count":571}
I20240501 08:11:21.774734 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling LogGCOp(6665039606404fe0a8e6c45674aadca4): free 1049101 bytes of WAL
I20240501 08:11:21.776655 26194 log_reader.cc:386] T 6665039606404fe0a8e6c45674aadca4: removed 1 log segments from log reader
I20240501 08:11:21.776779 26194 log.cc:1076] T 6665039606404fe0a8e6c45674aadca4 P ef205d178a5e41058e69ab8cc77cb795: Deleting log segment in path: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/wals/6665039606404fe0a8e6c45674aadca4/wal-000000001 (ops 1-737)
I20240501 08:11:21.777413 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: LogGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240501 08:11:21.777966 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014704
I20240501 08:11:21.872006 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.050s user 0.433s sys 0.034s Metrics: {"bytes_written":4054,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":24927,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":102445,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2539,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7895,"thread_start_us":24785,"threads_started":1,"update_count":603}
I20240501 08:11:21.876298 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling LogGCOp(6665039606404fe0a8e6c45674aadca4): free 1048883 bytes of WAL
I20240501 08:11:21.876713 26325 log_reader.cc:386] T 6665039606404fe0a8e6c45674aadca4: removed 1 log segments from log reader
I20240501 08:11:21.876842 26325 log.cc:1076] T 6665039606404fe0a8e6c45674aadca4 P a60f7f39696f4125b1fe90f5ac68cae1: Deleting log segment in path: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal/wals/6665039606404fe0a8e6c45674aadca4/wal-000000001 (ops 1-737)
I20240501 08:11:21.877501 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: LogGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20240501 08:11:21.878038 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014704
I20240501 08:11:22.439122 26607 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:22.484323 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801233746833408 Inserted count: 37750
I20240501 08:11:22.612481 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:23.121855 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 2.052s user 0.991s sys 0.022s Metrics: {"bytes_written":1689,"cfile_cache_hit":12,"cfile_cache_hit_bytes":45894,"cfile_cache_miss":78,"cfile_cache_miss_bytes":1562328,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":107850,"dirs.run_cpu_time_us":883,"dirs.run_wall_time_us":291504,"drs_written":1,"lbm_read_time_us":4360,"lbm_reads_lt_1ms":114,"lbm_write_time_us":28160,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":65,"num_input_rowsets":2,"peak_mem_usage":779875,"reinsert_count":0,"rows_written":23709,"thread_start_us":110067,"threads_started":5,"update_count":232}
I20240501 08:11:23.124009 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.004475
I20240501 08:11:23.458703 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.325s user 0.104s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2513,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":81,"lbm_read_time_us":438,"lbm_reads_lt_1ms":24,"lbm_write_time_us":1471,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"update_count":232}
I20240501 08:11:23.460309 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 442858 bytes on disk
I20240501 08:11:23.461151 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 08:11:23.489889 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.709s user 0.899s sys 0.047s Metrics: {"bytes_written":444386,"cfile_cache_hit":11,"cfile_cache_hit_bytes":824,"cfile_cache_miss":79,"cfile_cache_miss_bytes":1631064,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":10899,"dirs.run_cpu_time_us":1565,"dirs.run_wall_time_us":155739,"drs_written":1,"lbm_read_time_us":14082,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":113,"lbm_write_time_us":34153,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":62,"num_input_rowsets":2,"peak_mem_usage":795417,"rows_written":24048,"thread_start_us":26160,"threads_started":3}
I20240501 08:11:23.491020 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 452066 bytes on disk
I20240501 08:11:23.496642 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 08:11:23.560480 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.682s user 0.948s sys 0.077s Metrics: {"bytes_written":445932,"cfile_cache_hit":11,"cfile_cache_hit_bytes":824,"cfile_cache_miss":79,"cfile_cache_miss_bytes":1632108,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":15523,"dirs.run_cpu_time_us":1104,"dirs.run_wall_time_us":49206,"drs_written":1,"lbm_read_time_us":27448,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":114,"lbm_write_time_us":16713,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":63,"mutex_wait_us":699,"num_input_rowsets":2,"peak_mem_usage":796488,"rows_written":24080,"thread_start_us":64957,"threads_started":3}
I20240501 08:11:23.561854 26434 maintenance_manager.cc:419] P a60f7f39696f4125b1fe90f5ac68cae1: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 452730 bytes on disk
I20240501 08:11:23.562750 26325 maintenance_manager.cc:643] P a60f7f39696f4125b1fe90f5ac68cae1: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
F20240501 08:11:24.148351 26607 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f37123d6dcd google::LogMessage::Fail() at ??:0
@ 0x7f37123dab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f37123d67cc google::LogMessage::Flush() at ??:0
@ 0x7f37123d7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3714fffc80 kudu::operator-() at ??:0
@ 0x7f3721c8014a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f3721c4a5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f3721c4a336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f3721bf72d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f37246c4e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f37246d39fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f37246d3862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f371e10902a std::function<>::operator()() at ??:0
@ 0x7f371e1017e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f371e0fc7f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f371e0fa927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f371e138089 handle_request at ??:0
@ 0x7f371e13a76b worker_thread at ??:0
@ 0x7f371844b6db start_thread at ??:0
@ 0x7f371128971f clone at ??:0
W20240501 08:11:24.434430 26568 connection.cc:535] client connection to 127.25.118.66:35327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:24.434921 26568 meta_cache.cc:302] tablet 7f42407c614045e7bc0dbef2b85e2511: replica a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:24.435304 26453 connection.cc:535] client connection to 127.25.118.66:35327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:24.435511 26453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:24.439699 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:24.446424 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:24.452479 26217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.453006 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.465740 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.465759 26475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.487041 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.489907 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
I20240501 08:11:24.493932 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:24.499303 26475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.499303 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.522877 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.534144 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.534144 26475 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.545840 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.565708 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.575213 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.583461 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.593972 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.622356 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.631070 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.638924 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.651572 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.682145 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.697731 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.705312 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.721988 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.752933 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.769516 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.776984 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.797597 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.834708 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.854112 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.863965 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.885619 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.921741 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.947122 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.954895 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:24.980952 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.020628 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.045222 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.053038 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.082832 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.125133 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.157352 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.162655 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.194343 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.243145 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.271447 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.280715 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.312439 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.364010 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.399749 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.406229 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.445075 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.450224 26568 meta_cache.cc:302] tablet 7f42407c614045e7bc0dbef2b85e2511: replica a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327) has failed: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111)
I20240501 08:11:25.491026 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:25.500128 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.534703 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.547348 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
W20240501 08:11:25.583022 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
I20240501 08:11:25.631752 26791 raft_consensus.cc:482] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a60f7f39696f4125b1fe90f5ac68cae1)
I20240501 08:11:25.632009 26791 raft_consensus.cc:504] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:25.632953 26791 leader_election.cc:288] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65:40247), a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327)
I20240501 08:11:25.633621 26256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "ce059656f8cb4229845d41328f2957d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 812 } } ignore_live_leader: false dest_uuid: "ef205d178a5e41058e69ab8cc77cb795" is_pre_election: true
W20240501 08:11:25.634678 26453 leader_election.cc:334] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111)
I20240501 08:11:25.634820 26453 leader_election.cc:302] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce059656f8cb4229845d41328f2957d9; no voters: a60f7f39696f4125b1fe90f5ac68cae1, ef205d178a5e41058e69ab8cc77cb795
I20240501 08:11:25.635126 26791 raft_consensus.cc:2726] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:11:25.644464 26216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57030: Illegal state: replica ef205d178a5e41058e69ab8cc77cb795 is not leader of this config: current role FOLLOWER
I20240501 08:11:25.652640 26830 raft_consensus.cc:482] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a60f7f39696f4125b1fe90f5ac68cae1)
I20240501 08:11:25.652875 26830 raft_consensus.cc:504] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:25.654438 26830 leader_election.cc:288] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327), ce059656f8cb4229845d41328f2957d9 (127.25.118.67:45999)
W20240501 08:11:25.662487 26192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111)
W20240501 08:11:25.665491 26192 leader_election.cc:334] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111)
I20240501 08:11:25.669073 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "ef205d178a5e41058e69ab8cc77cb795" candidate_term: 2 candidate_status { last_received { term: 1 index: 812 } } ignore_live_leader: false dest_uuid: "ce059656f8cb4229845d41328f2957d9" is_pre_election: true
I20240501 08:11:25.669399 26518 raft_consensus.cc:2454] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef205d178a5e41058e69ab8cc77cb795 in term 1.
I20240501 08:11:25.669968 26193 leader_election.cc:302] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce059656f8cb4229845d41328f2957d9, ef205d178a5e41058e69ab8cc77cb795; no voters: a60f7f39696f4125b1fe90f5ac68cae1
I20240501 08:11:25.670400 26830 raft_consensus.cc:2781] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:11:25.670527 26830 raft_consensus.cc:482] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a60f7f39696f4125b1fe90f5ac68cae1)
I20240501 08:11:25.670683 26830 raft_consensus.cc:3037] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:25.672619 26830 raft_consensus.cc:504] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:25.673288 26830 leader_election.cc:288] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 election: Requested vote from peers a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327), ce059656f8cb4229845d41328f2957d9 (127.25.118.67:45999)
I20240501 08:11:25.673835 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f42407c614045e7bc0dbef2b85e2511" candidate_uuid: "ef205d178a5e41058e69ab8cc77cb795" candidate_term: 2 candidate_status { last_received { term: 1 index: 812 } } ignore_live_leader: false dest_uuid: "ce059656f8cb4229845d41328f2957d9"
I20240501 08:11:25.674065 26518 raft_consensus.cc:3037] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:11:25.674963 26192 leader_election.cc:334] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111)
I20240501 08:11:25.676246 26518 raft_consensus.cc:2454] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef205d178a5e41058e69ab8cc77cb795 in term 2.
I20240501 08:11:25.676678 26193 leader_election.cc:302] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce059656f8cb4229845d41328f2957d9, ef205d178a5e41058e69ab8cc77cb795; no voters: a60f7f39696f4125b1fe90f5ac68cae1
I20240501 08:11:25.677129 26830 raft_consensus.cc:2781] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:25.677592 26830 raft_consensus.cc:686] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [term 2 LEADER]: Becoming Leader. State: Replica: ef205d178a5e41058e69ab8cc77cb795, State: Running, Role: LEADER
I20240501 08:11:25.678158 26830 consensus_queue.cc:237] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 812, Committed index: 812, Last appended: 1.812, Last appended by leader: 812, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } }
I20240501 08:11:25.682318 26114 catalog_manager.cc:5484] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 reported cstate change: term changed from 1 to 2, leader changed from a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66) to ef205d178a5e41058e69ab8cc77cb795 (127.25.118.65). New cstate: current_term: 2 leader_uuid: "ef205d178a5e41058e69ab8cc77cb795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef205d178a5e41058e69ab8cc77cb795" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 40247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60f7f39696f4125b1fe90f5ac68cae1" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:25.690196 26518 raft_consensus.cc:1261] T 7f42407c614045e7bc0dbef2b85e2511 P ce059656f8cb4229845d41328f2957d9 [term 2 FOLLOWER]: Refusing update from remote peer ef205d178a5e41058e69ab8cc77cb795: Log matching property violated. Preceding OpId in replica: term: 1 index: 812. Preceding OpId from leader: term: 2 index: 814. (index mismatch)
I20240501 08:11:25.691161 26830 consensus_queue.cc:1035] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce059656f8cb4229845d41328f2957d9" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 813, Last known committed idx: 812, Time since last communication: 0.000s
W20240501 08:11:25.692023 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:25.692694 26478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60480: Illegal state: replica ce059656f8cb4229845d41328f2957d9 is not leader of this config: current role FOLLOWER
I20240501 08:11:26.108639 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:26.567750 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:26.956737 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:27.471989 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(eef0447a12844339b7e999cb340d96e6): perf score=1.000000
I20240501 08:11:27.505664 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(eef0447a12844339b7e999cb340d96e6): perf score=1.000000
I20240501 08:11:27.572710 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:28.038244 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.566s user 0.426s sys 0.052s Metrics: {"bytes_written":2422,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":79721,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1737,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3080,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7642,"thread_start_us":79604,"threads_started":1,"update_count":350}
I20240501 08:11:28.039536 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(7f42407c614045e7bc0dbef2b85e2511): perf score=1.000000
I20240501 08:11:28.052656 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.547s user 0.428s sys 0.000s Metrics: {"bytes_written":2558,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6401,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":103117,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2255,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7663,"thread_start_us":6561,"threads_started":1,"update_count":371}
I20240501 08:11:28.053680 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(7f42407c614045e7bc0dbef2b85e2511): perf score=1.000000
W20240501 08:11:28.342190 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:28.583400 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:28.639240 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(7f42407c614045e7bc0dbef2b85e2511) complete. Timing: real 0.585s user 0.410s sys 0.057s Metrics: {"bytes_written":143614,"cfile_init":1,"dirs.queue_time_us":12476,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":2773,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2067,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7687,"thread_start_us":12365,"threads_started":1}
I20240501 08:11:28.640204 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(7f42407c614045e7bc0dbef2b85e2511): 145779 bytes on disk
I20240501 08:11:28.640821 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(7f42407c614045e7bc0dbef2b85e2511) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 08:11:28.641580 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(eef0447a12844339b7e999cb340d96e6): 142481 bytes on disk
I20240501 08:11:28.644107 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 08:11:28.646167 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(7f42407c614045e7bc0dbef2b85e2511) complete. Timing: real 0.605s user 0.415s sys 0.057s Metrics: {"bytes_written":143614,"cfile_init":1,"dirs.queue_time_us":6318,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":5923,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2070,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7687,"thread_start_us":6449,"threads_started":1}
I20240501 08:11:28.646311 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling MajorDeltaCompactionOp(eef0447a12844339b7e999cb340d96e6): perf score=0.020740
I20240501 08:11:28.647017 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling MajorDeltaCompactionOp(eef0447a12844339b7e999cb340d96e6): perf score=0.019680
I20240501 08:11:28.740701 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: MajorDeltaCompactionOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.093s user 0.041s sys 0.028s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3234,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1092,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1817,"lbm_read_time_us":5565,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":22,"lbm_write_time_us":314,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":27252,"threads_started":1,"update_count":350}
I20240501 08:11:28.741694 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(7f42407c614045e7bc0dbef2b85e2511): 145779 bytes on disk
I20240501 08:11:28.740960 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: MajorDeltaCompactionOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.094s user 0.050s sys 0.012s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3403,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":3938,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":5994,"lbm_read_time_us":380,"lbm_reads_lt_1ms":20,"lbm_write_time_us":298,"lbm_writes_lt_1ms":8,"mutex_wait_us":2,"reinsert_count":0,"spinlock_wait_cycles":4992,"thread_start_us":32267,"threads_started":1,"update_count":371}
I20240501 08:11:28.742511 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(7f42407c614045e7bc0dbef2b85e2511) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 08:11:28.743363 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
W20240501 08:11:28.883965 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:28.993382 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling FlushMRSOp(6665039606404fe0a8e6c45674aadca4): perf score=1.000000
I20240501 08:11:29.253979 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.510s user 0.414s sys 0.048s Metrics: {"bytes_written":5725,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1017,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3198,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7910,"update_count":863}
I20240501 08:11:29.255312 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling LogGCOp(6665039606404fe0a8e6c45674aadca4): free 1048927 bytes of WAL
I20240501 08:11:29.255743 26455 log_reader.cc:386] T 6665039606404fe0a8e6c45674aadca4: removed 1 log segments from log reader
I20240501 08:11:29.255896 26455 log.cc:1076] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9: Deleting log segment in path: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal/wals/6665039606404fe0a8e6c45674aadca4/wal-000000001 (ops 1-735)
I20240501 08:11:29.256615 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: LogGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240501 08:11:29.257210 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(eef0447a12844339b7e999cb340d96e6): 142207 bytes on disk
I20240501 08:11:29.257826 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(eef0447a12844339b7e999cb340d96e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:11:29.258626 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 134012 bytes on disk
I20240501 08:11:29.260325 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:11:29.261269 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014681
W20240501 08:11:29.472165 26453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 08:11:29.477106 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:29.690960 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: FlushMRSOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.697s user 0.422s sys 0.020s Metrics: {"bytes_written":143584,"cfile_init":1,"dirs.queue_time_us":6529,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":15433,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2436,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7796,"spinlock_wait_cycles":160640,"thread_start_us":6645,"threads_started":1}
I20240501 08:11:29.692337 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4): perf score=1.014572
I20240501 08:11:30.408774 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:30.808663 26192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 08:11:30.809772 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:30.936163 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.674s user 1.265s sys 0.031s Metrics: {"bytes_written":581353,"cfile_cache_hit":12,"cfile_cache_hit_bytes":964,"cfile_cache_miss":96,"cfile_cache_miss_bytes":2169822,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":13180,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":975,"drs_written":1,"lbm_read_time_us":5366,"lbm_reads_lt_1ms":132,"lbm_write_time_us":8166,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":75,"num_input_rowsets":2,"peak_mem_usage":1049006,"rows_written":31619,"thread_start_us":13081,"threads_started":1}
I20240501 08:11:30.937386 26564 maintenance_manager.cc:419] P ce059656f8cb4229845d41328f2957d9: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 599898 bytes on disk
I20240501 08:11:30.938814 26455 maintenance_manager.cc:643] P ce059656f8cb4229845d41328f2957d9: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:11:31.087857 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: CompactRowSetsOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 1.395s user 1.183s sys 0.029s Metrics: {"bytes_written":792,"cfile_cache_hit":11,"cfile_cache_hit_bytes":46841,"cfile_cache_miss":91,"cfile_cache_miss_bytes":2143087,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":24600,"dirs.run_cpu_time_us":1629,"dirs.run_wall_time_us":14652,"drs_written":1,"lbm_read_time_us":5172,"lbm_reads_lt_1ms":123,"lbm_write_time_us":10894,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":79,"mutex_wait_us":580,"num_input_rowsets":2,"peak_mem_usage":1054910,"reinsert_count":0,"rows_written":31844,"spinlock_wait_cycles":83328,"thread_start_us":25736,"threads_started":7,"update_count":93}
I20240501 08:11:31.089089 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4): 603711 bytes on disk
I20240501 08:11:31.089761 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: UndoDeltaBlockGCOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:11:31.090721 26303 maintenance_manager.cc:419] P ef205d178a5e41058e69ab8cc77cb795: Scheduling MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4): perf score=0.001561
I20240501 08:11:31.291481 26194 maintenance_manager.cc:643] P ef205d178a5e41058e69ab8cc77cb795: MajorDeltaCompactionOp(6665039606404fe0a8e6c45674aadca4) complete. Timing: real 0.200s user 0.122s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1109,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":111,"lbm_read_time_us":363,"lbm_reads_lt_1ms":20,"lbm_write_time_us":279,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":93}
W20240501 08:11:31.345777 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:31.417639 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:31.869410 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:11:31.880115 26192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.118.66:35327, user_credentials={real_user=slave}} blocked reactor thread for 52704.4us
I20240501 08:11:32.448144 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801274634805248 Inserted count: 54400
I20240501 08:11:32.448292 26073 linked_list-test-util.h:421] Finished inserting list. Added 54400 in chain
I20240501 08:11:32.448357 26073 linked_list-test-util.h:422] Last entries inserted had keys:
I20240501 08:11:32.448423 26073 linked_list-test-util.h:424] 0: -8661795982752415744
I20240501 08:11:32.448478 26073 linked_list-test-util.h:424] 1: -7097020113598545919
I20240501 08:11:32.448562 26073 linked_list-test-util.h:424] 2: -5532103506956255230
I20240501 08:11:32.448618 26073 linked_list-test-util.h:424] 3: -3967327637802385405
I20240501 08:11:32.448699 26073 linked_list-test-util.h:424] 4: 7382537059797041156
I20240501 08:11:32.448746 26073 linked_list-test-util.h:424] 5: 8947594403927621637
I20240501 08:11:32.448824 26073 linked_list-test-util.h:424] 6: -7934373800628060154
I20240501 08:11:32.448882 26073 linked_list-test-util.h:424] 7: -6369457193985769465
I20240501 08:11:32.448951 26073 linked_list-test-util.h:424] 8: 4980548241101946888
I20240501 08:11:32.449009 26073 linked_list-test-util.h:424] 9: 6545183372767526921
I20240501 08:11:32.449070 26073 linked_list-test-util.h:424] 10: 8110240716898107402
I20240501 08:11:32.449118 26073 linked_list-test-util.h:424] 11: -8771868225145864181
I20240501 08:11:32.449174 26073 linked_list-test-util.h:424] 12: 2578137209941852172
I20240501 08:11:32.449219 26073 linked_list-test-util.h:424] 13: 4143053816584142861
I20240501 08:11:32.449275 26073 linked_list-test-util.h:424] 14: 5707829685738012686
I20240501 08:11:32.449319 26073 linked_list-test-util.h:424] 15: 7272887029868593167
I20240501 08:11:32.449375 26073 linked_list-test-util.h:424] 16: 176007653758468112
I20240501 08:11:32.449421 26073 linked_list-test-util.h:424] 17: 1740642785424048145
I20240501 08:11:32.449486 26073 linked_list-test-util.h:424] 18: 3305700129554628626
I20240501 08:11:32.449530 26073 linked_list-test-util.h:424] 19: 4870475998708498451
I20240501 08:11:32.449596 26073 linked_list-test-util.h:424] 20: -2226403377401626604
I20240501 08:11:32.449651 26073 linked_list-test-util.h:424] 21: -661346033271046123
I20240501 08:11:32.449714 26073 linked_list-test-util.h:424] 22: 903289098394533910
I20240501 08:11:32.449765 26073 linked_list-test-util.h:424] 23: 2468346442525114391
I20240501 08:11:32.449828 26073 linked_list-test-util.h:424] 24: -4628532933585010664
I20240501 08:11:32.449884 26073 linked_list-test-util.h:424] 25: -3063757064431140839
I20240501 08:11:32.449946 26073 linked_list-test-util.h:424] 26: -1498840457788850150
I20240501 08:11:32.449998 26073 linked_list-test-util.h:424] 27: -8595579096410685413
I20240501 08:11:32.450062 26073 linked_list-test-util.h:424] 28: -7030943964745105380
I20240501 08:11:32.450119 26073 linked_list-test-util.h:424] 29: -5465886620614524899
I20240501 08:11:32.450179 26073 linked_list-test-util.h:424] 30: -3901110751460655074
I20240501 08:11:32.450228 26073 linked_list-test-util.h:424] 31: 7448753946138771487
I20240501 08:11:32.450290 26073 linked_list-test-util.h:424] 32: 9013811290269351968
I20240501 08:11:32.450340 26073 linked_list-test-util.h:424] 33: -7868297651774619615
I20240501 08:11:32.450403 26073 linked_list-test-util.h:424] 34: -6303240307644039134
I20240501 08:11:32.450472 26073 linked_list-test-util.h:424] 35: 5046624389955387427
I20240501 08:11:32.450541 26073 linked_list-test-util.h:424] 36: 6611400259109257252
I20240501 08:11:32.450611 26073 linked_list-test-util.h:424] 37: 8176316865751547941
I20240501 08:11:32.450675 26073 linked_list-test-util.h:424] 38: -8705651338804133850
I20240501 08:11:32.450726 26073 linked_list-test-util.h:424] 39: 2644213358795292711
I20240501 08:11:32.450788 26073 linked_list-test-util.h:424] 40: 4209270702925873192
I20240501 08:11:32.450841 26073 linked_list-test-util.h:424] 41: 5773905834591453225
I20240501 08:11:32.450904 26073 linked_list-test-util.h:424] 42: 7338963178722033706
I20240501 08:11:32.450954 26073 linked_list-test-util.h:424] 43: 242083802611908651
I20240501 08:11:32.451017 26073 linked_list-test-util.h:424] 44: 1806859671765778476
I20240501 08:11:32.451067 26073 linked_list-test-util.h:424] 45: 3371917015896358957
I20240501 08:11:32.451129 26073 linked_list-test-util.h:424] 46: 4936552147561938990
I20240501 08:11:32.451180 26073 linked_list-test-util.h:424] 47: -2160186491059896273
I20240501 08:11:32.451242 26073 linked_list-test-util.h:424] 48: -595269884417605584
I20240501 08:11:32.451292 26073 linked_list-test-util.h:424] 49: 969505984736264241
I20240501 08:11:32.503970 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551072445529 usec, L: 1 Expected Rows: 20300
I20240501 08:11:32.531812 26073 linked_list-test-util.h:528] Calling callback on tserver ce059656f8cb4229845d41328f2957d9
I20240501 08:11:32.900084 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:33.133982 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:33.228952 26073 linked_list-test-util.h:767] Done collecting results (20300 rows in 697.317ms)
I20240501 08:11:33.274700 26073 linked_list-test-util.h:556] Snapshot: P: 1714551072445529 usec, L: 1 verified. Result: OK
I20240501 08:11:33.279146 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551082457723 usec, L: 1 Expected Rows: 37750
I20240501 08:11:33.298835 26073 linked_list-test-util.h:528] Calling callback on tserver ce059656f8cb4229845d41328f2957d9
I20240501 08:11:33.610340 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:33.765201 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:33.992337 26073 linked_list-test-util.h:767] Done collecting results (37750 rows in 693.667ms)
I20240501 08:11:34.073848 26073 linked_list-test-util.h:556] Snapshot: P: 1714551082457723 usec, L: 1 verified. Result: OK
I20240501 08:11:34.077478 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551092440138 usec, L: 1 Expected Rows: 54400
I20240501 08:11:34.086795 26073 linked_list-test-util.h:528] Calling callback on tserver ce059656f8cb4229845d41328f2957d9
W20240501 08:11:34.477671 26453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:11:34.478950 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:34.699484 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:35.282035 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240501 08:11:35.284051 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:35.490212 26073 linked_list-test-util.h:767] Done collecting results (54400 rows in 1403.56ms)
I20240501 08:11:35.560554 26073 linked_list-test-util.h:556] Snapshot: P: 1714551092440138 usec, L: 1 verified. Result: OK
I20240501 08:11:35.563597 26073 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 54400
W20240501 08:11:35.818230 26192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:11:36.337379 26453 consensus_peers.cc:485] T eef0447a12844339b7e999cb340d96e6 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:11:36.397670 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:36.903728 26073 linked_list-test-util.h:767] Done collecting results (54400 rows in 1321.75ms)
I20240501 08:11:36.923463 26609 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:36.968245 26073 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240501 08:11:36.968739 26073 linked_list-test-util.h:692] Successfully verified 54400 rows
W20240501 08:11:36.970420 26453 consensus_peers.cc:485] T 6665039606404fe0a8e6c45674aadca4 P ce059656f8cb4229845d41328f2957d9 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs:
TabletServer: a60f7f39696f4125b1fe90f5ac68cae1, Rpc address: host: "127.25.118.66" port: 35327 (Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111))
I20240501 08:11:36.987972 26073 webui_checker.cc:88] shutting down CURL thread
W20240501 08:11:37.849826 26192 consensus_peers.cc:485] T 7f42407c614045e7bc0dbef2b85e2511 P ef205d178a5e41058e69ab8cc77cb795 -> Peer a60f7f39696f4125b1fe90f5ac68cae1 (127.25.118.66:35327): Couldn't send request to peer a60f7f39696f4125b1fe90f5ac68cae1. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.66:35327: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:11:37.858209 26637 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:11:37.862639 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 26175
I20240501 08:11:37.863997 26297 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:38.076187 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 26175
I20240501 08:11:38.124331 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 26436
I20240501 08:11:38.125423 26558 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:38.362286 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 26436
I20240501 08:11:38.411486 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 26085
I20240501 08:11:38.412757 26146 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:38.532583 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 26085
2024-05-01T08:11:38Z chronyd exiting
I20240501 08:11:38.571062 26073 test_util.cc:175] -----------------------------------------------
I20240501 08:11:38.571180 26073 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714551059952006-26073-0
[ FAILED ] LinkedListTest.TestLoadAndVerify (38548 ms)
[ RUN ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240501 08:11:38.573132 26073 test_util.cc:268] Using random seed: 154009908
I20240501 08:11:38.575129 26073 linked_list-test.cc:100] Linked List Test Configuration:
I20240501 08:11:38.575208 26073 linked_list-test.cc:101] --------------
I20240501 08:11:38.575273 26073 linked_list-test.cc:102] 50 chains
I20240501 08:11:38.575313 26073 linked_list-test.cc:103] 3 tablets
I20240501 08:11:38.575371 26073 linked_list-test.cc:104] Mutations on
I20240501 08:11:38.575413 26073 linked_list-test.cc:105] --------------
I20240501 08:11:38.575572 26073 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:38.575616 26073 ts_itest-base.cc:113] --------------
I20240501 08:11:38.575675 26073 ts_itest-base.cc:114] 3 tablet servers
I20240501 08:11:38.575718 26073 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:38.575776 26073 ts_itest-base.cc:116] --------------
2024-05-01T08:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:38Z Disabled control of system clock
I20240501 08:11:38.608366 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.118.126:45825
--webserver_interface=127.25.118.126
--webserver_port=0
--builtin_ntp_servers=127.25.118.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.118.126:45825
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240501 08:11:38.968582 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:38.968938 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:38.969074 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:38.976709 26934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:38.976848 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:38.976930 26934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:38.977005 26934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:38.987004 26934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.118.126:45825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.118.126:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.25.118.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:38.988445 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:38.990343 26934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:11:39.002883 26934 server_base.cc:1034] running on GCE node
W20240501 08:11:39.003552 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:39.004184 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:39.004205 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:39.005239 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:39.006889 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:39.008172 26934 hybrid_clock.cc:648] HybridClock initialized: now 1714551099008098 us; error 48 us; skew 500 ppm
I20240501 08:11:39.008559 26934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:39.016360 26934 webserver.cc:460] Webserver started at http://127.25.118.126:39021/ using document root <none> and password file <none>
I20240501 08:11:39.017241 26934 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:39.017405 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:39.017732 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:39.020885 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/instance:
uuid: "dc2e0d0618e94f828c945327169a92fa"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.021728 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal/instance:
uuid: "dc2e0d0618e94f828c945327169a92fa"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.027731 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240501 08:11:39.031692 26948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:39.033241 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:39.033442 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
uuid: "dc2e0d0618e94f828c945327169a92fa"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.033646 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:39.056723 26934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:39.057619 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:39.057924 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:39.076020 26934 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.126:45825
I20240501 08:11:39.076038 26999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.126:45825 every 8 connection(s)
I20240501 08:11:39.077546 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/data/info.pb
I20240501 08:11:39.081604 27000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.082263 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 26934
I20240501 08:11:39.082446 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/master-0/wal/instance
I20240501 08:11:39.094336 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa: Bootstrap starting.
I20240501 08:11:39.098140 27000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.099263 27000 log.cc:824] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:39.102217 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa: No bootstrap required, opened a new log
I20240501 08:11:39.108883 27000 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } }
I20240501 08:11:39.109308 27000 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.109427 27000 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2e0d0618e94f828c945327169a92fa, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.110088 27000 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } }
I20240501 08:11:39.110313 27000 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:39.110433 27000 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:39.110603 27000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.112844 27000 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } }
I20240501 08:11:39.113646 27000 leader_election.cc:302] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2e0d0618e94f828c945327169a92fa; no voters:
I20240501 08:11:39.114287 27000 leader_election.cc:288] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:39.114426 27005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:39.115002 27005 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [term 1 LEADER]: Becoming Leader. State: Replica: dc2e0d0618e94f828c945327169a92fa, State: Running, Role: LEADER
I20240501 08:11:39.115613 27005 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } }
I20240501 08:11:39.116438 27000 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:39.118445 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2e0d0618e94f828c945327169a92fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } } }
I20240501 08:11:39.118449 27007 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2e0d0618e94f828c945327169a92fa. Latest consensus state: current_term: 1 leader_uuid: "dc2e0d0618e94f828c945327169a92fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e0d0618e94f828c945327169a92fa" member_type: VOTER last_known_addr { host: "127.25.118.126" port: 45825 } } }
I20240501 08:11:39.118906 27007 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:39.119113 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:39.126546 27011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:39.132225 27011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:39.143013 27011 catalog_manager.cc:1347] Generated new cluster ID: 58f49ac2fb1847cd86a08d874915441b
I20240501 08:11:39.143158 27011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:39.165071 27011 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:39.166497 27011 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:39.180459 27011 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dc2e0d0618e94f828c945327169a92fa: Generated new TSK 0
I20240501 08:11:39.181407 27011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:39.199434 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.65:0
--local_ip_for_outbound_sockets=127.25.118.65
--webserver_interface=127.25.118.65
--webserver_port=0
--tserver_master_addrs=127.25.118.126:45825
--builtin_ntp_servers=127.25.118.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240501 08:11:39.596774 27024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:39.597100 27024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:39.597244 27024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:39.605038 27024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:39.605284 27024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.65
I20240501 08:11:39.614739 27024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.25.118.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.25.118.126:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.65
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:39.616210 27024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:39.618158 27024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:39.632129 27030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:39.632710 27032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:39.633220 27029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:39.634230 27024 server_base.cc:1034] running on GCE node
I20240501 08:11:39.635072 27024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:39.636114 27024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:39.637353 27024 hybrid_clock.cc:648] HybridClock initialized: now 1714551099637280 us; error 49 us; skew 500 ppm
I20240501 08:11:39.637775 27024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:39.640377 27024 webserver.cc:460] Webserver started at http://127.25.118.65:41277/ using document root <none> and password file <none>
I20240501 08:11:39.641168 27024 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:39.641289 27024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:39.641572 27024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:39.644470 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/instance:
uuid: "5878e2c169cf42878d1611f5b6b7016b"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.645282 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/instance:
uuid: "5878e2c169cf42878d1611f5b6b7016b"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.651170 27024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:11:39.655161 27038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:39.656847 27024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:39.657049 27024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
uuid: "5878e2c169cf42878d1611f5b6b7016b"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:11:39.657269 27024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:39.674624 27024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:39.675519 27024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:39.675946 27024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:39.677047 27024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:39.678737 27024 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:39.678825 27024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:39.678946 27024 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:39.678998 27024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:39.709932 27024 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.65:44917
I20240501 08:11:39.709970 27150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.65:44917 every 8 connection(s)
I20240501 08:11:39.711336 27024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
I20240501 08:11:39.717880 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 27024
I20240501 08:11:39.718109 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/instance
I20240501 08:11:39.721428 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.66:0
--local_ip_for_outbound_sockets=127.25.118.66
--webserver_interface=127.25.118.66
--webserver_port=0
--tserver_master_addrs=127.25.118.126:45825
--builtin_ntp_servers=127.25.118.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240501 08:11:39.725905 27151 heartbeater.cc:347] Connected to a master server at 127.25.118.126:45825
I20240501 08:11:39.726307 27151 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:39.727291 27151 heartbeater.cc:511] Master 127.25.118.126:45825 requested a full tablet report, sending...
I20240501 08:11:39.729758 26965 ts_manager.cc:194] Registered new tserver with Master: 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917)
I20240501 08:11:39.731437 26965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.65:37519
W20240501 08:11:40.067764 27155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:40.068125 27155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:40.068284 27155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:40.076854 27155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:40.077108 27155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.66
I20240501 08:11:40.086501 27155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.25.118.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.25.118.126:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.66
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:40.087985 27155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:40.089833 27155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:40.103560 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.103901 27163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.104816 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:40.105566 27155 server_base.cc:1034] running on GCE node
I20240501 08:11:40.106261 27155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:40.107309 27155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:40.108577 27155 hybrid_clock.cc:648] HybridClock initialized: now 1714551100108503 us; error 46 us; skew 500 ppm
I20240501 08:11:40.108922 27155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:40.111446 27155 webserver.cc:460] Webserver started at http://127.25.118.66:42455/ using document root <none> and password file <none>
I20240501 08:11:40.112231 27155 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:40.112370 27155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:40.112677 27155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:40.115540 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/instance:
uuid: "964ab967903d4dd0b4767ebc638c0d3f"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.116325 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal/instance:
uuid: "964ab967903d4dd0b4767ebc638c0d3f"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.122061 27155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:40.126264 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.128011 27155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 08:11:40.128219 27155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
uuid: "964ab967903d4dd0b4767ebc638c0d3f"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.128432 27155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:40.164786 27155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:40.165727 27155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:40.166043 27155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:40.167201 27155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:40.168886 27155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:40.168972 27155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.169104 27155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:40.169157 27155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.200227 27155 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.66:34389
I20240501 08:11:40.200273 27281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.66:34389 every 8 connection(s)
I20240501 08:11:40.201546 27155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/data/info.pb
I20240501 08:11:40.206871 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 27155
I20240501 08:11:40.207083 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-1/wal/instance
I20240501 08:11:40.210880 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.67:0
--local_ip_for_outbound_sockets=127.25.118.67
--webserver_interface=127.25.118.67
--webserver_port=0
--tserver_master_addrs=127.25.118.126:45825
--builtin_ntp_servers=127.25.118.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240501 08:11:40.219178 27282 heartbeater.cc:347] Connected to a master server at 127.25.118.126:45825
I20240501 08:11:40.219559 27282 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:40.220654 27282 heartbeater.cc:511] Master 127.25.118.126:45825 requested a full tablet report, sending...
I20240501 08:11:40.222380 26965 ts_manager.cc:194] Registered new tserver with Master: 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389)
I20240501 08:11:40.223227 26965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.66:49721
W20240501 08:11:40.556289 27285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:40.556613 27285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:40.556742 27285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:40.564791 27285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:40.565039 27285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.67
I20240501 08:11:40.574623 27285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.25.118.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.25.118.126:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.67
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:40.576077 27285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:40.577956 27285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:40.591406 27290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.591960 27293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.592594 27291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:40.593348 27285 server_base.cc:1034] running on GCE node
I20240501 08:11:40.594105 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:40.595243 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:40.596483 27285 hybrid_clock.cc:648] HybridClock initialized: now 1714551100596395 us; error 65 us; skew 500 ppm
I20240501 08:11:40.596817 27285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:40.599444 27285 webserver.cc:460] Webserver started at http://127.25.118.67:43779/ using document root <none> and password file <none>
I20240501 08:11:40.600241 27285 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:40.600379 27285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:40.600689 27285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:40.603518 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/instance:
uuid: "5e910684d5f343f7ae57a665819a2a3b"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.604293 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal/instance:
uuid: "5e910684d5f343f7ae57a665819a2a3b"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.610013 27285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:11:40.613965 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.615646 27285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:40.615852 27285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
uuid: "5e910684d5f343f7ae57a665819a2a3b"
format_stamp: "Formatted at 2024-05-01 08:11:40 on dist-test-slave-2ds8"
I20240501 08:11:40.616051 27285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:40.642004 27285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:40.642966 27285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:40.643278 27285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:40.644281 27285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:40.645887 27285 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:40.645969 27285 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.646090 27285 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:40.646142 27285 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.678442 27285 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.67:38035
I20240501 08:11:40.678483 27411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.67:38035 every 8 connection(s)
I20240501 08:11:40.679853 27285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/data/info.pb
I20240501 08:11:40.685843 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 27285
I20240501 08:11:40.686043 26073 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-2/wal/instance
I20240501 08:11:40.693946 27412 heartbeater.cc:347] Connected to a master server at 127.25.118.126:45825
I20240501 08:11:40.694265 27412 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:40.695179 27412 heartbeater.cc:511] Master 127.25.118.126:45825 requested a full tablet report, sending...
I20240501 08:11:40.696772 26965 ts_manager.cc:194] Registered new tserver with Master: 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035)
I20240501 08:11:40.697561 26965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.67:50811
I20240501 08:11:40.702212 26073 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 08:11:40.719990 26965 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44248:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240501 08:11:40.726343 26965 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:11:40.735313 27151 heartbeater.cc:503] Master 127.25.118.126:45825 was elected leader, sending a full tablet report...
I20240501 08:11:40.766865 27347 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba4084f0fba47a99a0a1a86fb34738f (DEFAULT_TABLE table=TestTable [id=6359bcdd2e17444ba5684d114b013d2a]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240501 08:11:40.766789 27086 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba4084f0fba47a99a0a1a86fb34738f (DEFAULT_TABLE table=TestTable [id=6359bcdd2e17444ba5684d114b013d2a]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240501 08:11:40.768131 27086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba4084f0fba47a99a0a1a86fb34738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.768160 27347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba4084f0fba47a99a0a1a86fb34738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.772717 27217 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba4084f0fba47a99a0a1a86fb34738f (DEFAULT_TABLE table=TestTable [id=6359bcdd2e17444ba5684d114b013d2a]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240501 08:11:40.774384 27217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba4084f0fba47a99a0a1a86fb34738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.779810 27430 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Bootstrap starting.
I20240501 08:11:40.783591 27430 tablet_bootstrap.cc:654] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.785205 27430 log.cc:824] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:40.785493 27431 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Bootstrap starting.
I20240501 08:11:40.788759 27433 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Bootstrap starting.
I20240501 08:11:40.789287 27430 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: No bootstrap required, opened a new log
I20240501 08:11:40.789664 27430 ts_tablet_manager.cc:1367] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:11:40.790019 27431 tablet_bootstrap.cc:654] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.791405 27431 log.cc:824] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:40.791843 27433 tablet_bootstrap.cc:654] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.792973 27433 log.cc:824] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:40.795426 27431 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: No bootstrap required, opened a new log
I20240501 08:11:40.795751 27431 ts_tablet_manager.cc:1367] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20240501 08:11:40.796613 27433 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: No bootstrap required, opened a new log
I20240501 08:11:40.796865 27433 ts_tablet_manager.cc:1367] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240501 08:11:40.798214 27430 raft_consensus.cc:348] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.798691 27430 raft_consensus.cc:374] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.798836 27430 raft_consensus.cc:729] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5878e2c169cf42878d1611f5b6b7016b, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.799607 27430 consensus_queue.cc:260] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.803423 27433 raft_consensus.cc:348] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.804003 27433 raft_consensus.cc:374] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.804152 27433 raft_consensus.cc:729] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964ab967903d4dd0b4767ebc638c0d3f, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.804152 27430 ts_tablet_manager.cc:1398] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20240501 08:11:40.804996 27433 consensus_queue.cc:260] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.806257 27282 heartbeater.cc:503] Master 127.25.118.126:45825 was elected leader, sending a full tablet report...
I20240501 08:11:40.806227 27431 raft_consensus.cc:348] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.806809 27431 raft_consensus.cc:374] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.806893 27433 ts_tablet_manager.cc:1398] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20240501 08:11:40.806912 27431 raft_consensus.cc:729] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e910684d5f343f7ae57a665819a2a3b, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.807734 27431 consensus_queue.cc:260] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.809182 27412 heartbeater.cc:503] Master 127.25.118.126:45825 was elected leader, sending a full tablet report...
I20240501 08:11:40.809752 27431 ts_tablet_manager.cc:1398] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
W20240501 08:11:40.931643 27413 tablet.cc:2369] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:40.953691 27283 tablet.cc:2369] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:40.962819 27437 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:40.963116 27437 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
W20240501 08:11:40.964048 27152 tablet.cc:2369] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:40.965394 27437 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917)
I20240501 08:11:40.972443 27106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba4084f0fba47a99a0a1a86fb34738f" candidate_uuid: "964ab967903d4dd0b4767ebc638c0d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5878e2c169cf42878d1611f5b6b7016b" is_pre_election: true
I20240501 08:11:40.972630 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba4084f0fba47a99a0a1a86fb34738f" candidate_uuid: "964ab967903d4dd0b4767ebc638c0d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e910684d5f343f7ae57a665819a2a3b" is_pre_election: true
I20240501 08:11:40.972970 27106 raft_consensus.cc:2454] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964ab967903d4dd0b4767ebc638c0d3f in term 0.
I20240501 08:11:40.973145 27367 raft_consensus.cc:2454] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964ab967903d4dd0b4767ebc638c0d3f in term 0.
I20240501 08:11:40.973716 27173 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b, 964ab967903d4dd0b4767ebc638c0d3f; no voters:
I20240501 08:11:40.974169 27437 raft_consensus.cc:2781] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:40.974349 27437 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:40.974472 27437 raft_consensus.cc:3037] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.976649 27437 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.977376 27437 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [CANDIDATE]: Term 1 election: Requested vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917)
I20240501 08:11:40.977919 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba4084f0fba47a99a0a1a86fb34738f" candidate_uuid: "964ab967903d4dd0b4767ebc638c0d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e910684d5f343f7ae57a665819a2a3b"
I20240501 08:11:40.978070 27106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba4084f0fba47a99a0a1a86fb34738f" candidate_uuid: "964ab967903d4dd0b4767ebc638c0d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5878e2c169cf42878d1611f5b6b7016b"
I20240501 08:11:40.978197 27367 raft_consensus.cc:3037] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.978276 27106 raft_consensus.cc:3037] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.981334 27106 raft_consensus.cc:2454] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964ab967903d4dd0b4767ebc638c0d3f in term 1.
I20240501 08:11:40.981329 27367 raft_consensus.cc:2454] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964ab967903d4dd0b4767ebc638c0d3f in term 1.
I20240501 08:11:40.981885 27173 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b, 964ab967903d4dd0b4767ebc638c0d3f; no voters:
I20240501 08:11:40.982228 27437 raft_consensus.cc:2781] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:40.982685 27437 raft_consensus.cc:686] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [term 1 LEADER]: Becoming Leader. State: Replica: 964ab967903d4dd0b4767ebc638c0d3f, State: Running, Role: LEADER
I20240501 08:11:40.983369 27437 consensus_queue.cc:237] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:11:40.988610 26964 catalog_manager.cc:5484] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66). New cstate: current_term: 1 leader_uuid: "964ab967903d4dd0b4767ebc638c0d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:41.052712 26073 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 08:11:41.055496 26073 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5878e2c169cf42878d1611f5b6b7016b to finish bootstrapping
I20240501 08:11:41.065176 26073 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 964ab967903d4dd0b4767ebc638c0d3f to finish bootstrapping
I20240501 08:11:41.073444 26073 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5e910684d5f343f7ae57a665819a2a3b to finish bootstrapping
I20240501 08:11:41.082639 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 27024
I20240501 08:11:41.083884 27146 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:41.169116 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 27024
I20240501 08:11:41.188449 26073 linked_list-test-util.h:401] Running for: 30.000s
I20240501 08:11:41.188585 26073 linked_list-test-util.h:402] Sampling every 10000000 us
I20240501 08:11:41.188673 26073 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
W20240501 08:11:41.212803 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111)
I20240501 08:11:41.213404 27367 raft_consensus.cc:1261] T 8ba4084f0fba47a99a0a1a86fb34738f P 5e910684d5f343f7ae57a665819a2a3b [term 1 FOLLOWER]: Refusing update from remote peer 964ab967903d4dd0b4767ebc638c0d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:41.214541 27441 consensus_queue.cc:1035] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:41.214977 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:41.223481 27447 mvcc.cc:205] Tried to move back new op lower bound from 7022801310557302784 to 7022801309631995904. Current Snapshot: MvccSnapshot[applied={T|T < 7022801310557302784}]
W20240501 08:11:43.242705 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:45.615641 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:46.564426 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 08:11:46.967845 27453 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 30 ops behind the committed index
W20240501 08:11:47.866815 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:50.003891 27453 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 305 ops behind the committed index [suppressed 5 similar messages]
W20240501 08:11:50.542352 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:51.192135 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801351418671104 Inserted count: 44550
W20240501 08:11:51.707214 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:11:53.147517 27453 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 577 ops behind the committed index [suppressed 5 similar messages]
W20240501 08:11:53.155061 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:55.729990 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:11:56.207402 27453 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 848 ops behind the committed index [suppressed 5 similar messages]
W20240501 08:11:57.068933 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:11:58.130640 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:59.410334 27540 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 1136 ops behind the committed index [suppressed 6 similar messages]
W20240501 08:12:00.418526 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:12:01.190129 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801392372359168 Inserted count: 88750
I20240501 08:12:02.494259 27453 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 1426 ops behind the committed index [suppressed 5 similar messages]
W20240501 08:12:02.503118 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:12:02.980206 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:12:05.502174 27573 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 1704 ops behind the committed index [suppressed 5 similar messages]
W20240501 08:12:05.510447 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:12:07.576059 27173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:12:08.032127 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:12:08.997975 27573 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 2014 ops behind the committed index [suppressed 6 similar messages]
W20240501 08:12:10.471082 27173 consensus_peers.cc:485] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f -> Peer 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917): Couldn't send request to peer 5878e2c169cf42878d1611f5b6b7016b. Status: Network error: Client connection negotiation failed: client connection to 127.25.118.65:44917: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:12:11.189024 26073 linked_list-test-util.h:417] Sample at HT timestamp: 7022801433327120384 Inserted count: 133600
I20240501 08:12:11.189247 26073 linked_list-test-util.h:421] Finished inserting list. Added 133600 in chain
I20240501 08:12:11.189363 26073 linked_list-test-util.h:422] Last entries inserted had keys:
I20240501 08:12:11.189448 26073 linked_list-test-util.h:424] 0: -505235017183002624
I20240501 08:12:11.189512 26073 linked_list-test-util.h:424] 1: -2309654691342909439
I20240501 08:12:11.189584 26073 linked_list-test-util.h:424] 2: -4114215102991106046
I20240501 08:12:11.189630 26073 linked_list-test-util.h:424] 3: -5918775514639302653
I20240501 08:12:11.189697 26073 linked_list-test-util.h:424] 4: -8228570943470501884
I20240501 08:12:11.189752 26073 linked_list-test-util.h:424] 5: 8413612718590853125
I20240501 08:12:11.189821 26073 linked_list-test-util.h:424] 6: 6609052306942656518
I20240501 08:12:11.189878 26073 linked_list-test-util.h:424] 7: 4804351157806039047
I20240501 08:12:11.189954 26073 linked_list-test-util.h:424] 8: 2494837203951550472
I20240501 08:12:11.190011 26073 linked_list-test-util.h:424] 9: 690276792303353865
I20240501 08:12:11.190078 26073 linked_list-test-util.h:424] 10: -1114565094321553398
I20240501 08:12:11.190138 26073 linked_list-test-util.h:424] 11: -2919125505969750005
I20240501 08:12:11.190204 26073 linked_list-test-util.h:424] 12: -5228639459824238580
I20240501 08:12:11.190258 26073 linked_list-test-util.h:424] 13: -7033199871472435187
I20240501 08:12:11.190325 26073 linked_list-test-util.h:424] 14: -8837901020609052658
I20240501 08:12:11.190380 26073 linked_list-test-util.h:424] 15: 7804282641452302351
I20240501 08:12:11.190448 26073 linked_list-test-util.h:424] 16: 5494768687597813776
I20240501 08:12:11.190503 26073 linked_list-test-util.h:424] 17: 3689926800972906513
I20240501 08:12:11.190572 26073 linked_list-test-util.h:424] 18: 1885366389324709906
I20240501 08:12:11.190655 26073 linked_list-test-util.h:424] 19: 80805977676513299
I20240501 08:12:11.190724 26073 linked_list-test-util.h:424] 20: -2228989451154685932
I20240501 08:12:11.190790 26073 linked_list-test-util.h:424] 21: -4033409125314592747
I20240501 08:12:11.190874 26073 linked_list-test-util.h:424] 22: -5837969536962789354
I20240501 08:12:11.190940 26073 linked_list-test-util.h:424] 23: -7642529948610985961
I20240501 08:12:11.191008 26073 linked_list-test-util.h:424] 24: 8494418696267366424
I20240501 08:12:11.191063 26073 linked_list-test-util.h:424] 25: 6689858284619169817
I20240501 08:12:11.191140 26073 linked_list-test-util.h:424] 26: 4885297872970973210
I20240501 08:12:11.191200 26073 linked_list-test-util.h:424] 27: 2575502444139773979
I20240501 08:12:11.191260 26073 linked_list-test-util.h:424] 28: 771082769979867164
I20240501 08:12:11.191315 26073 linked_list-test-util.h:424] 29: -1033477641668329443
I20240501 08:12:11.191376 26073 linked_list-test-util.h:424] 30: -2838038053316526050
I20240501 08:12:11.191432 26073 linked_list-test-util.h:424] 31: -5147833482147725281
I20240501 08:12:11.191505 26073 linked_list-test-util.h:424] 32: -6952393893795921888
I20240501 08:12:11.191557 26073 linked_list-test-util.h:424] 33: -8756954305444118495
I20240501 08:12:11.191624 26073 linked_list-test-util.h:424] 34: 7885370094105526306
I20240501 08:12:11.191679 26073 linked_list-test-util.h:424] 35: 5575574665274327075
I20240501 08:12:11.191737 26073 linked_list-test-util.h:424] 36: 3771014253626130468
I20240501 08:12:11.191792 26073 linked_list-test-util.h:424] 37: 1966453841977933861
I20240501 08:12:11.191864 26073 linked_list-test-util.h:424] 38: 161893430329737254
I20240501 08:12:11.191918 26073 linked_list-test-util.h:424] 39: -2147901998501461977
I20240501 08:12:11.191958 26073 linked_list-test-util.h:424] 40: -3952462410149658584
I20240501 08:12:11.191991 26073 linked_list-test-util.h:424] 41: -5756882084309565399
I20240501 08:12:11.192032 26073 linked_list-test-util.h:424] 42: -7561442495957762006
I20240501 08:12:11.192065 26073 linked_list-test-util.h:424] 43: 8575506148920590379
I20240501 08:12:11.192103 26073 linked_list-test-util.h:424] 44: 6770945737272393772
I20240501 08:12:11.192135 26073 linked_list-test-util.h:424] 45: 4966385325624197165
I20240501 08:12:11.192170 26073 linked_list-test-util.h:424] 46: 3161824913976000558
I20240501 08:12:11.192203 26073 linked_list-test-util.h:424] 47: 852029485144801327
I20240501 08:12:11.192238 26073 linked_list-test-util.h:424] 48: -952390189015105488
I20240501 08:12:11.192267 26073 linked_list-test-util.h:424] 49: -2756950600663302095
I20240501 08:12:11.255880 26073 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6CbDo8/build/asan/bin/kudu
/tmp/dist-test-task6CbDo8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.118.65:44917
--local_ip_for_outbound_sockets=127.25.118.65
--tserver_master_addrs=127.25.118.126:45825
--webserver_port=41277
--webserver_interface=127.25.118.65
--builtin_ntp_servers=127.25.118.84:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240501 08:12:11.608732 27615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:11.609031 27615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:11.609135 27615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:11.617273 27615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:11.617501 27615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.118.65
I20240501 08:12:11.627008 27615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.118.84:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.118.65:44917
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.25.118.65
--webserver_port=41277
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.25.118.126:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.118.65
--log_dir=/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:11.628414 27615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:11.630190 27615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:11.643896 27620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:11.643965 27621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:11.644580 27623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:11.645113 27615 server_base.cc:1034] running on GCE node
I20240501 08:12:11.645781 27615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:11.646823 27615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:11.648085 27615 hybrid_clock.cc:648] HybridClock initialized: now 1714551131647994 us; error 59 us; skew 500 ppm
I20240501 08:12:11.648487 27615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:11.656939 27615 webserver.cc:460] Webserver started at http://127.25.118.65:41277/ using document root <none> and password file <none>
I20240501 08:12:11.657749 27615 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:11.657892 27615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:11.663517 27615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:12:11.666682 27629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:11.668339 27615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:11.668506 27615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
uuid: "5878e2c169cf42878d1611f5b6b7016b"
format_stamp: "Formatted at 2024-05-01 08:11:39 on dist-test-slave-2ds8"
I20240501 08:12:11.669425 27615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:11.687435 27615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:11.688390 27615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:11.688694 27615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:11.689745 27615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:11.692445 27636 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 08:12:11.699911 27615 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 08:12:11.700066 27615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20240501 08:12:11.700201 27615 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 08:12:11.703357 27615 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 08:12:11.703467 27615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:11.703963 27636 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Bootstrap starting.
I20240501 08:12:11.726042 27636 log.cc:824] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:11.730086 27636 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:11.730636 27636 log.cc:1225] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/wal/wals/8ba4084f0fba47a99a0a1a86fb34738f.recovery-1714551131730386
I20240501 08:12:11.730782 27636 tablet_bootstrap.cc:492] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Bootstrap complete.
I20240501 08:12:11.731529 27636 ts_tablet_manager.cc:1367] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.011s
I20240501 08:12:11.744194 27615 rpc_server.cc:307] RPC server started. Bound to: 127.25.118.65:44917
I20240501 08:12:11.744216 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.118.65:44917 every 8 connection(s)
I20240501 08:12:11.746084 27615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6CbDo8/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714551059952006-26073-0/linked-list-cluster/ts-0/data/info.pb
I20240501 08:12:11.745154 27636 raft_consensus.cc:348] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:11.746331 27636 raft_consensus.cc:729] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5878e2c169cf42878d1611f5b6b7016b, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.747327 27636 consensus_queue.cc:260] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:11.749135 27636 ts_tablet_manager.cc:1398] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20240501 08:12:11.750797 26073 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6CbDo8/build/asan/bin/kudu as pid 27615
I20240501 08:12:11.759794 27744 heartbeater.cc:347] Connected to a master server at 127.25.118.126:45825
I20240501 08:12:11.760154 27744 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:11.761138 27744 heartbeater.cc:511] Master 127.25.118.126:45825 requested a full tablet report, sending...
I20240501 08:12:11.764142 26960 ts_manager.cc:194] Re-registered known tserver with Master: 5878e2c169cf42878d1611f5b6b7016b (127.25.118.65:44917)
I20240501 08:12:11.765540 26960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.118.65:54421
I20240501 08:12:11.768842 27744 heartbeater.cc:503] Master 127.25.118.126:45825 was elected leader, sending a full tablet report...
I20240501 08:12:11.779247 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 0.0
I20240501 08:12:11.882973 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 0.0
W20240501 08:12:11.998116 27745 tablet.cc:2369] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:12.086566 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 0.0
I20240501 08:12:12.167793 27610 consensus_queue.cc:786] T 8ba4084f0fba47a99a0a1a86fb34738f P 964ab967903d4dd0b4767ebc638c0d3f [LEADER]: Peer 5878e2c169cf42878d1611f5b6b7016b is lagging by at least 2211 ops behind the committed index [suppressed 6 similar messages]
I20240501 08:12:12.395363 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 1.492
I20240501 08:12:12.873793 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 1.1457
I20240501 08:12:13.087594 27753 mvcc.cc:205] Tried to move back new op lower bound from 7022801332898721792 to 7022801309631995904. Current Snapshot: MvccSnapshot[applied={T|T < 7022801331596492800 or (T in {7022801331602411520,7022801331698868224,7022801331744247808,7022801331790094336,7022801331834843136,7022801331883061248,7022801331926142976,7022801332022267904,7022801332065234944,7022801332158738432,7022801332111937536,7022801332207038464,7022801332252938240,7022801332298596352,7022801332342685696,7022801332387610624,7022801332469940224,7022801332430802944,7022801332508844032,7022801332554010624,7022801332593545216,7022801332669759488,7022801332745084928,7022801332632387584})}]
I20240501 08:12:13.377275 26073 cluster_itest_util.cc:256] Not converged past 2672 yet: 1.2703 1.2703 1.2435
I20240501 08:12:13.981176 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 27155
I20240501 08:12:13.982329 27277 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:14.286898 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 27155
I20240501 08:12:14.354522 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 27285
I20240501 08:12:14.355916 27407 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:14.659793 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 27285
I20240501 08:12:14.719280 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551111186199 usec, L: 1 Expected Rows: 44550
I20240501 08:12:14.726945 26073 meta_cache.cc:1510] marking tablet server 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) as failed
W20240501 08:12:14.727196 26073 meta_cache.cc:302] tablet 8ba4084f0fba47a99a0a1a86fb34738f: replica 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
I20240501 08:12:14.728708 26073 meta_cache.cc:1510] marking tablet server 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389) as failed
I20240501 08:12:14.815042 26073 linked_list-test-util.h:528] Calling callback on tserver 5878e2c169cf42878d1611f5b6b7016b
I20240501 08:12:15.387818 27767 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 964ab967903d4dd0b4767ebc638c0d3f)
I20240501 08:12:15.391901 27767 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:15.425628 27767 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389)
W20240501 08:12:15.435051 27633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.66:34389: connect: Connection refused (error 111)
W20240501 08:12:15.437464 27632 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035): Network error: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
W20240501 08:12:15.438764 27633 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389): Network error: Client connection negotiation failed: client connection to 127.25.118.66:34389: connect: Connection refused (error 111)
I20240501 08:12:15.440052 27633 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b; no voters: 5e910684d5f343f7ae57a665819a2a3b, 964ab967903d4dd0b4767ebc638c0d3f
I20240501 08:12:15.440542 27767 raft_consensus.cc:2726] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:16.831027 26073 linked_list-test-util.h:767] Done collecting results (44550 rows in 2016.15ms)
I20240501 08:12:16.912194 26073 linked_list-test-util.h:556] Snapshot: P: 1714551111186199 usec, L: 1 verified. Result: OK
I20240501 08:12:16.915655 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551121184658 usec, L: 1 Expected Rows: 88750
I20240501 08:12:16.917560 26073 meta_cache.cc:1510] marking tablet server 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) as failed
W20240501 08:12:16.917702 26073 meta_cache.cc:302] tablet 8ba4084f0fba47a99a0a1a86fb34738f: replica 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
I20240501 08:12:16.918962 26073 meta_cache.cc:1510] marking tablet server 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389) as failed
I20240501 08:12:16.931077 26073 linked_list-test-util.h:528] Calling callback on tserver 5878e2c169cf42878d1611f5b6b7016b
I20240501 08:12:17.376243 27772 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 964ab967903d4dd0b4767ebc638c0d3f)
I20240501 08:12:17.376420 27772 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:17.377063 27772 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389)
W20240501 08:12:17.379091 27632 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035): Network error: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
W20240501 08:12:17.379303 27633 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389): Network error: Client connection negotiation failed: client connection to 127.25.118.66:34389: connect: Connection refused (error 111)
I20240501 08:12:17.379395 27633 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b; no voters: 5e910684d5f343f7ae57a665819a2a3b, 964ab967903d4dd0b4767ebc638c0d3f
I20240501 08:12:17.379640 27772 raft_consensus.cc:2726] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:18.897264 26073 linked_list-test-util.h:767] Done collecting results (88750 rows in 1966.31ms)
I20240501 08:12:19.004256 26073 linked_list-test-util.h:556] Snapshot: P: 1714551121184658 usec, L: 1 verified. Result: OK
I20240501 08:12:19.007413 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 1714551131183379 usec, L: 1 Expected Rows: 133600
I20240501 08:12:19.009001 26073 meta_cache.cc:1510] marking tablet server 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) as failed
W20240501 08:12:19.009136 26073 meta_cache.cc:302] tablet 8ba4084f0fba47a99a0a1a86fb34738f: replica 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
I20240501 08:12:19.010064 26073 meta_cache.cc:1510] marking tablet server 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389) as failed
I20240501 08:12:19.019400 26073 linked_list-test-util.h:528] Calling callback on tserver 5878e2c169cf42878d1611f5b6b7016b
I20240501 08:12:20.597880 27777 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 964ab967903d4dd0b4767ebc638c0d3f)
I20240501 08:12:20.598038 27777 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:20.598743 27777 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389)
W20240501 08:12:20.599613 27632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 08:12:20.600787 27632 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035): Network error: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
W20240501 08:12:20.600991 27633 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389): Network error: Client connection negotiation failed: client connection to 127.25.118.66:34389: connect: Connection refused (error 111)
I20240501 08:12:20.601089 27633 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b; no voters: 5e910684d5f343f7ae57a665819a2a3b, 964ab967903d4dd0b4767ebc638c0d3f
I20240501 08:12:20.601336 27777 raft_consensus.cc:2726] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:21.228660 26073 linked_list-test-util.h:767] Done collecting results (133600 rows in 2209.38ms)
I20240501 08:12:21.392977 26073 linked_list-test-util.h:556] Snapshot: P: 1714551131183379 usec, L: 1 verified. Result: OK
I20240501 08:12:21.396334 26073 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 133600
I20240501 08:12:21.398151 26073 meta_cache.cc:1510] marking tablet server 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) as failed
W20240501 08:12:21.398286 26073 meta_cache.cc:302] tablet 8ba4084f0fba47a99a0a1a86fb34738f: replica 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
I20240501 08:12:21.399335 26073 meta_cache.cc:1510] marking tablet server 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389) as failed
I20240501 08:12:21.403882 26073 linked_list-test-util.h:528] Calling callback on tserver 5878e2c169cf42878d1611f5b6b7016b
I20240501 08:12:22.813864 27782 raft_consensus.cc:482] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 964ab967903d4dd0b4767ebc638c0d3f)
I20240501 08:12:22.814054 27782 raft_consensus.cc:504] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e910684d5f343f7ae57a665819a2a3b" member_type: VOTER last_known_addr { host: "127.25.118.67" port: 38035 } } peers { permanent_uuid: "5878e2c169cf42878d1611f5b6b7016b" member_type: VOTER last_known_addr { host: "127.25.118.65" port: 44917 } } peers { permanent_uuid: "964ab967903d4dd0b4767ebc638c0d3f" member_type: VOTER last_known_addr { host: "127.25.118.66" port: 34389 } }
I20240501 08:12:22.814798 27782 leader_election.cc:288] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035), 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389)
W20240501 08:12:22.818411 27632 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e910684d5f343f7ae57a665819a2a3b (127.25.118.67:38035): Network error: Client connection negotiation failed: client connection to 127.25.118.67:38035: connect: Connection refused (error 111)
W20240501 08:12:22.818632 27633 leader_election.cc:334] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 964ab967903d4dd0b4767ebc638c0d3f (127.25.118.66:34389): Network error: Client connection negotiation failed: client connection to 127.25.118.66:34389: connect: Connection refused (error 111)
I20240501 08:12:22.818748 27633 leader_election.cc:302] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5878e2c169cf42878d1611f5b6b7016b; no voters: 5e910684d5f343f7ae57a665819a2a3b, 964ab967903d4dd0b4767ebc638c0d3f
I20240501 08:12:22.818995 27782 raft_consensus.cc:2726] T 8ba4084f0fba47a99a0a1a86fb34738f P 5878e2c169cf42878d1611f5b6b7016b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:23.533582 26073 linked_list-test-util.h:767] Done collecting results (133600 rows in 2129.81ms)
I20240501 08:12:23.694985 26073 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240501 08:12:23.695813 26073 linked_list-test-util.h:692] Successfully verified 133600 rows
I20240501 08:12:23.697199 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 27615
I20240501 08:12:23.704488 27739 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:23.929026 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 27615
I20240501 08:12:23.970270 26073 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6CbDo8/build/asan/bin/kudu pid 26934
I20240501 08:12:23.971503 26995 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:24.077744 26073 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6CbDo8/build/asan/bin/kudu with pid 26934
2024-05-01T08:12:24Z chronyd exiting
[ OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (45549 ms)
[----------] 2 tests from LinkedListTest (84097 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (84098 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] LinkedListTest.TestLoadAndVerify
1 FAILED TEST
I20240501 08:12:24.121737 26073 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~2 seconds ago