[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:30.873567 21452 test_util.cc:268] Using random seed: -618199116
I20240502 01:52:30.885090 21452 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 01:52:30.885188 21452 linked_list-test.cc:101] --------------
I20240502 01:52:30.885231 21452 linked_list-test.cc:102] 50 chains
I20240502 01:52:30.885277 21452 linked_list-test.cc:103] 3 tablets
I20240502 01:52:30.885327 21452 linked_list-test.cc:104] Mutations on
I20240502 01:52:30.885368 21452 linked_list-test.cc:105] --------------
I20240502 01:52:30.885576 21452 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:30.885627 21452 ts_itest-base.cc:113] --------------
I20240502 01:52:30.885676 21452 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:52:30.885721 21452 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:30.885759 21452 ts_itest-base.cc:116] --------------
2024-05-02T01:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:30Z Disabled control of system clock
I20240502 01:52:30.912565 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.243.62:42079
--webserver_interface=127.20.243.62
--webserver_port=0
--builtin_ntp_servers=127.20.243.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.243.62:42079
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:52:31.241241 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.241550 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.241655 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.248889 21465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:31.248999 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.249054 21465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:31.249116 21465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:31.258042 21465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.243.62:42079
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.243.62:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.20.243.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:31.259467 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:31.261224 21465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:31.272161 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.272310 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.272835 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:31.273041 21465 server_base.cc:1034] running on GCE node
I20240502 01:52:31.273852 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:31.275346 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:31.276582 21465 hybrid_clock.cc:648] HybridClock initialized: now 1714614751276521 us; error 44 us; skew 500 ppm
I20240502 01:52:31.276899 21465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:31.283051 21465 webserver.cc:460] Webserver started at http://127.20.243.62:36437/ using document root <none> and password file <none>
I20240502 01:52:31.283804 21465 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:31.283939 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:31.284248 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:31.287058 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/instance:
uuid: "265dd3086aa74dabb84ff297b9dd0370"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.287781 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal/instance:
uuid: "265dd3086aa74dabb84ff297b9dd0370"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.292889 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:31.296170 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.297446 21465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:31.297608 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
uuid: "265dd3086aa74dabb84ff297b9dd0370"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.297780 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:31.330209 21465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:31.331182 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:31.331499 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:31.350334 21465 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.62:42079
I20240502 01:52:31.350384 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.62:42079 every 8 connection(s)
I20240502 01:52:31.351907 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
I20240502 01:52:31.355861 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:31.361534 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 21465
I20240502 01:52:31.361847 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal/instance
I20240502 01:52:31.365258 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370: Bootstrap starting.
I20240502 01:52:31.369406 21532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:31.370466 21532 log.cc:824] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:31.373059 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370: No bootstrap required, opened a new log
I20240502 01:52:31.380632 21532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } }
I20240502 01:52:31.381047 21532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:31.381178 21532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265dd3086aa74dabb84ff297b9dd0370, State: Initialized, Role: FOLLOWER
I20240502 01:52:31.381963 21532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } }
I20240502 01:52:31.382251 21532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:31.382426 21532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:31.382616 21532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:31.385725 21532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } }
I20240502 01:52:31.386262 21532 leader_election.cc:302] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265dd3086aa74dabb84ff297b9dd0370; no voters:
I20240502 01:52:31.387410 21532 leader_election.cc:288] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:31.387521 21537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:31.389796 21532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:31.390450 21537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [term 1 LEADER]: Becoming Leader. State: Replica: 265dd3086aa74dabb84ff297b9dd0370, State: Running, Role: LEADER
I20240502 01:52:31.391148 21537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } }
I20240502 01:52:31.394393 21539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265dd3086aa74dabb84ff297b9dd0370. Latest consensus state: current_term: 1 leader_uuid: "265dd3086aa74dabb84ff297b9dd0370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } } }
I20240502 01:52:31.394778 21539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:31.395201 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "265dd3086aa74dabb84ff297b9dd0370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265dd3086aa74dabb84ff297b9dd0370" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 42079 } } }
I20240502 01:52:31.395535 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:31.396515 21548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:31.401024 21548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:31.409919 21548 catalog_manager.cc:1347] Generated new cluster ID: a051118d925540dfa9d65b168ec30cfe
I20240502 01:52:31.410039 21548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:31.433622 21548 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:31.435163 21548 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:31.446362 21548 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 265dd3086aa74dabb84ff297b9dd0370: Generated new TSK 0
I20240502 01:52:31.447350 21548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:31.460883 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.1:0
--local_ip_for_outbound_sockets=127.20.243.1
--webserver_interface=127.20.243.1
--webserver_port=0
--tserver_master_addrs=127.20.243.62:42079
--builtin_ntp_servers=127.20.243.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 01:52:31.801892 21556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.802248 21556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.802456 21556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.810370 21556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.810521 21556 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:52:31.810590 21556 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:52:31.810686 21556 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:52:31.810804 21556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.1
I20240502 01:52:31.820631 21556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.243.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.243.62:42079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.1
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:31.822079 21556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:31.823944 21556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:31.836153 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.836249 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.836856 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:31.837038 21556 server_base.cc:1034] running on GCE node
I20240502 01:52:31.837654 21556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:31.838618 21556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:31.839895 21556 hybrid_clock.cc:648] HybridClock initialized: now 1714614751839833 us; error 45 us; skew 500 ppm
I20240502 01:52:31.840210 21556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:31.842500 21556 webserver.cc:460] Webserver started at http://127.20.243.1:36191/ using document root <none> and password file <none>
I20240502 01:52:31.843252 21556 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:31.843372 21556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:31.843681 21556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:31.846385 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/instance:
uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.847128 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/instance:
uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.852334 21556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240502 01:52:31.855700 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.856976 21556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:31.857141 21556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-l6m7"
I20240502 01:52:31.857342 21556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:31.871584 21556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:31.872396 21556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:31.872671 21556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:31.873569 21556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:31.874995 21556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:31.875115 21556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.875236 21556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:31.875317 21556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.907927 21556 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.1:37113
I20240502 01:52:31.908005 21686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.1:37113 every 8 connection(s)
I20240502 01:52:31.909682 21556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:52:31.918970 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 21556
I20240502 01:52:31.919193 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/instance
I20240502 01:52:31.923174 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.2:0
--local_ip_for_outbound_sockets=127.20.243.2
--webserver_interface=127.20.243.2
--webserver_port=0
--tserver_master_addrs=127.20.243.62:42079
--builtin_ntp_servers=127.20.243.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 01:52:31.924288 21687 heartbeater.cc:347] Connected to a master server at 127.20.243.62:42079
I20240502 01:52:31.924649 21687 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:31.925769 21687 heartbeater.cc:511] Master 127.20.243.62:42079 requested a full tablet report, sending...
I20240502 01:52:31.928081 21497 ts_manager.cc:194] Registered new tserver with Master: 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113)
I20240502 01:52:31.929641 21497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.1:33537
W20240502 01:52:32.263557 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:32.263901 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:32.264068 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:32.271687 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:32.271884 21691 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:52:32.271965 21691 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:52:32.272071 21691 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:52:32.272188 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.2
I20240502 01:52:32.281636 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.20.243.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.243.62:42079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.2
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:32.283114 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.284910 21691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.297540 21697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.297479 21698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.297509 21700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.298152 21691 server_base.cc:1034] running on GCE node
I20240502 01:52:32.298914 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:32.299935 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:32.301156 21691 hybrid_clock.cc:648] HybridClock initialized: now 1714614752301107 us; error 41 us; skew 500 ppm
I20240502 01:52:32.301499 21691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.303908 21691 webserver.cc:460] Webserver started at http://127.20.243.2:33909/ using document root <none> and password file <none>
I20240502 01:52:32.304702 21691 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.304814 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.305152 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.307950 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/instance:
uuid: "6620bb8e459945eb8e042f7806df6cd5"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.308822 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal/instance:
uuid: "6620bb8e459945eb8e042f7806df6cd5"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.314260 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 01:52:32.317763 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.319144 21691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:32.319306 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
uuid: "6620bb8e459945eb8e042f7806df6cd5"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.319509 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.338106 21691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.339056 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:32.339358 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.340355 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.341877 21691 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.341981 21691 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.342100 21691 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.342176 21691 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.377511 21691 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.2:35255
I20240502 01:52:32.377564 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.2:35255 every 8 connection(s)
I20240502 01:52:32.378952 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
I20240502 01:52:32.385978 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 21691
I20240502 01:52:32.386168 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal/instance
I20240502 01:52:32.389532 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.3:0
--local_ip_for_outbound_sockets=127.20.243.3
--webserver_interface=127.20.243.3
--webserver_port=0
--tserver_master_addrs=127.20.243.62:42079
--builtin_ntp_servers=127.20.243.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 01:52:32.392558 21820 heartbeater.cc:347] Connected to a master server at 127.20.243.62:42079
I20240502 01:52:32.392938 21820 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.394007 21820 heartbeater.cc:511] Master 127.20.243.62:42079 requested a full tablet report, sending...
I20240502 01:52:32.395679 21497 ts_manager.cc:194] Registered new tserver with Master: 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255)
I20240502 01:52:32.396494 21497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.2:44103
W20240502 01:52:32.732263 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:32.732612 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:32.732803 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:32.740594 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:32.740804 21823 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 01:52:32.740873 21823 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 01:52:32.740981 21823 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 01:52:32.741107 21823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.3
I20240502 01:52:32.750190 21823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.20.243.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.20.243.62:42079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.3
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:32.751600 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.753383 21823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.765589 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.766146 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.766209 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.766783 21823 server_base.cc:1034] running on GCE node
I20240502 01:52:32.767386 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:32.768358 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:32.769570 21823 hybrid_clock.cc:648] HybridClock initialized: now 1714614752769524 us; error 33 us; skew 500 ppm
I20240502 01:52:32.769891 21823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.772300 21823 webserver.cc:460] Webserver started at http://127.20.243.3:40189/ using document root <none> and password file <none>
I20240502 01:52:32.773082 21823 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.773200 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.773545 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.776405 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/instance:
uuid: "22defba24bbd4eacb7349f0158b38522"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.777222 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal/instance:
uuid: "22defba24bbd4eacb7349f0158b38522"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.782894 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240502 01:52:32.786479 21837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.787899 21823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 01:52:32.788074 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
uuid: "22defba24bbd4eacb7349f0158b38522"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-l6m7"
I20240502 01:52:32.788283 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.811609 21823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.812525 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:32.812812 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.813791 21823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.815428 21823 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.815521 21823 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.815641 21823 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.815721 21823 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.849452 21823 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.3:41963
I20240502 01:52:32.849572 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.3:41963 every 8 connection(s)
I20240502 01:52:32.850881 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
I20240502 01:52:32.859928 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 21823
I20240502 01:52:32.860127 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal/instance
I20240502 01:52:32.863512 21953 heartbeater.cc:347] Connected to a master server at 127.20.243.62:42079
I20240502 01:52:32.863837 21953 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.864861 21953 heartbeater.cc:511] Master 127.20.243.62:42079 requested a full tablet report, sending...
I20240502 01:52:32.866142 21497 ts_manager.cc:194] Registered new tserver with Master: 22defba24bbd4eacb7349f0158b38522 (127.20.243.3:41963)
I20240502 01:52:32.866955 21497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.3:36777
I20240502 01:52:32.876695 21452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:52:32.898715 21497 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39738:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 01:52:32.905238 21497 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:52:32.932404 21687 heartbeater.cc:503] Master 127.20.243.62:42079 was elected leader, sending a full tablet report...
I20240502 01:52:32.936865 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 039bd9290f5e44169be2d8f53a9c87bc (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:52:32.936647 21618 tablet_service.cc:1467] Processing CreateTablet for tablet b448622701684a588baf8dde2bf57cb9 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:52:32.937880 21619 tablet_service.cc:1467] Processing CreateTablet for tablet 039bd9290f5e44169be2d8f53a9c87bc (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:52:32.937305 21886 tablet_service.cc:1467] Processing CreateTablet for tablet 039bd9290f5e44169be2d8f53a9c87bc (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 01:52:32.936678 21752 tablet_service.cc:1467] Processing CreateTablet for tablet 4c17d10e240b44bfbfc539fb22cdfda2 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:52:32.938607 21618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448622701684a588baf8dde2bf57cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.938778 21885 tablet_service.cc:1467] Processing CreateTablet for tablet b448622701684a588baf8dde2bf57cb9 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:52:32.939137 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c17d10e240b44bfbfc539fb22cdfda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.939201 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039bd9290f5e44169be2d8f53a9c87bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.939528 21884 tablet_service.cc:1467] Processing CreateTablet for tablet 4c17d10e240b44bfbfc539fb22cdfda2 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:52:32.940062 21617 tablet_service.cc:1467] Processing CreateTablet for tablet 4c17d10e240b44bfbfc539fb22cdfda2 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 01:52:32.940572 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c17d10e240b44bfbfc539fb22cdfda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.941980 21886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039bd9290f5e44169be2d8f53a9c87bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.936537 21753 tablet_service.cc:1467] Processing CreateTablet for tablet b448622701684a588baf8dde2bf57cb9 (DEFAULT_TABLE table=TestTable [id=630c857d521b40448a53f160e13ce504]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 01:52:32.942699 21884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c17d10e240b44bfbfc539fb22cdfda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.942795 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448622701684a588baf8dde2bf57cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.946727 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039bd9290f5e44169be2d8f53a9c87bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.950073 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448622701684a588baf8dde2bf57cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.969247 21972 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: Bootstrap starting.
I20240502 01:52:32.969982 21971 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Bootstrap starting.
I20240502 01:52:32.970779 21973 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: Bootstrap starting.
I20240502 01:52:32.973536 21972 tablet_bootstrap.cc:654] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.973703 21971 tablet_bootstrap.cc:654] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.974138 21973 tablet_bootstrap.cc:654] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.974690 21972 log.cc:824] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:32.975078 21971 log.cc:824] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:32.975394 21973 log.cc:824] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:32.977893 21973 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: No bootstrap required, opened a new log
I20240502 01:52:32.977988 21971 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: No bootstrap required, opened a new log
I20240502 01:52:32.977988 21972 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: No bootstrap required, opened a new log
I20240502 01:52:32.978185 21973 ts_tablet_manager.cc:1367] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 01:52:32.978274 21972 ts_tablet_manager.cc:1367] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:52:32.978274 21971 ts_tablet_manager.cc:1367] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 01:52:32.984850 21973 raft_consensus.cc:348] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:32.985392 21973 raft_consensus.cc:374] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.985530 21973 raft_consensus.cc:729] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee5f2a8853c40f88bc2a2a1ed479323, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.986382 21973 consensus_queue.cc:260] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:32.987561 21971 raft_consensus.cc:348] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:32.987561 21972 raft_consensus.cc:348] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:32.988014 21971 raft_consensus.cc:374] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.988014 21972 raft_consensus.cc:374] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.988117 21972 raft_consensus.cc:729] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6620bb8e459945eb8e042f7806df6cd5, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.988114 21971 raft_consensus.cc:729] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22defba24bbd4eacb7349f0158b38522, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.988237 21973 ts_tablet_manager.cc:1398] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20240502 01:52:32.988636 21973 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Bootstrap starting.
I20240502 01:52:32.988803 21971 consensus_queue.cc:260] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:32.988804 21972 consensus_queue.cc:260] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:32.990406 21973 tablet_bootstrap.cc:654] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.998281 21820 heartbeater.cc:503] Master 127.20.243.62:42079 was elected leader, sending a full tablet report...
I20240502 01:52:32.998561 21971 ts_tablet_manager.cc:1398] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20240502 01:52:32.998927 21973 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: No bootstrap required, opened a new log
I20240502 01:52:32.999035 21972 ts_tablet_manager.cc:1398] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5: Time spent starting tablet: real 0.021s user 0.018s sys 0.002s
I20240502 01:52:32.999123 21973 ts_tablet_manager.cc:1367] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240502 01:52:32.999217 21953 heartbeater.cc:503] Master 127.20.243.62:42079 was elected leader, sending a full tablet report...
I20240502 01:52:32.999608 21972 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: Bootstrap starting.
I20240502 01:52:33.000149 21973 raft_consensus.cc:348] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.000430 21973 raft_consensus.cc:374] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.000552 21973 raft_consensus.cc:729] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee5f2a8853c40f88bc2a2a1ed479323, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.000978 21971 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: Bootstrap starting.
I20240502 01:52:33.000854 21973 consensus_queue.cc:260] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.002378 21973 ts_tablet_manager.cc:1398] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:33.002691 21973 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323: Bootstrap starting.
I20240502 01:52:33.002878 21971 tablet_bootstrap.cc:654] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.004662 21973 tablet_bootstrap.cc:654] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.009663 21972 tablet_bootstrap.cc:654] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.009681 21973 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323: No bootstrap required, opened a new log
I20240502 01:52:33.009852 21973 ts_tablet_manager.cc:1367] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:52:33.010015 21971 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: No bootstrap required, opened a new log
I20240502 01:52:33.010169 21971 ts_tablet_manager.cc:1367] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240502 01:52:33.010732 21973 raft_consensus.cc:348] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.010999 21973 raft_consensus.cc:374] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.011085 21973 raft_consensus.cc:729] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee5f2a8853c40f88bc2a2a1ed479323, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.011162 21971 raft_consensus.cc:348] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.011368 21971 raft_consensus.cc:374] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.011482 21971 raft_consensus.cc:729] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22defba24bbd4eacb7349f0158b38522, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.011364 21973 consensus_queue.cc:260] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.011732 21971 consensus_queue.cc:260] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.012013 21973 ts_tablet_manager.cc:1398] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.012392 21971 ts_tablet_manager.cc:1398] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.011993 21972 tablet_bootstrap.cc:492] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: No bootstrap required, opened a new log
I20240502 01:52:33.012663 21971 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Bootstrap starting.
I20240502 01:52:33.012797 21972 ts_tablet_manager.cc:1367] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240502 01:52:33.013833 21972 raft_consensus.cc:348] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.014094 21972 raft_consensus.cc:374] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.014201 21972 raft_consensus.cc:729] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6620bb8e459945eb8e042f7806df6cd5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.014367 21971 tablet_bootstrap.cc:654] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.014456 21972 consensus_queue.cc:260] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.015089 21972 ts_tablet_manager.cc:1398] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.015400 21972 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: Bootstrap starting.
I20240502 01:52:33.017038 21972 tablet_bootstrap.cc:654] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.017112 21971 tablet_bootstrap.cc:492] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: No bootstrap required, opened a new log
I20240502 01:52:33.017261 21971 ts_tablet_manager.cc:1367] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 01:52:33.017859 21971 raft_consensus.cc:348] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.018072 21971 raft_consensus.cc:374] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.018163 21971 raft_consensus.cc:729] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22defba24bbd4eacb7349f0158b38522, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.018524 21971 consensus_queue.cc:260] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.019142 21971 ts_tablet_manager.cc:1398] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:33.019500 21972 tablet_bootstrap.cc:492] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: No bootstrap required, opened a new log
I20240502 01:52:33.019671 21972 ts_tablet_manager.cc:1367] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:52:33.020401 21972 raft_consensus.cc:348] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.020666 21972 raft_consensus.cc:374] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.020752 21972 raft_consensus.cc:729] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6620bb8e459945eb8e042f7806df6cd5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.021011 21972 consensus_queue.cc:260] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.021579 21972 ts_tablet_manager.cc:1398] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:33.044988 21978 raft_consensus.cc:482] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.045302 21978 raft_consensus.cc:504] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.047125 21978 leader_election.cc:288] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113), 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255)
I20240502 01:52:33.053975 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448622701684a588baf8dde2bf57cb9" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" is_pre_election: true
I20240502 01:52:33.054490 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448622701684a588baf8dde2bf57cb9" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6620bb8e459945eb8e042f7806df6cd5" is_pre_election: true
I20240502 01:52:33.054638 21640 raft_consensus.cc:2454] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 0.
I20240502 01:52:33.054991 21774 raft_consensus.cc:2454] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 0.
I20240502 01:52:33.055335 21840 leader_election.cc:302] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 3ee5f2a8853c40f88bc2a2a1ed479323; no voters:
I20240502 01:52:33.055722 21978 raft_consensus.cc:2781] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.055887 21978 raft_consensus.cc:482] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.056000 21978 raft_consensus.cc:3037] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.058010 21978 raft_consensus.cc:504] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.058706 21978 leader_election.cc:288] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113), 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255)
I20240502 01:52:33.059114 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448622701684a588baf8dde2bf57cb9" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
I20240502 01:52:33.059203 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448622701684a588baf8dde2bf57cb9" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6620bb8e459945eb8e042f7806df6cd5"
I20240502 01:52:33.059301 21640 raft_consensus.cc:3037] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.059399 21774 raft_consensus.cc:3037] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.061488 21640 raft_consensus.cc:2454] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 1.
I20240502 01:52:33.061563 21774 raft_consensus.cc:2454] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 1.
I20240502 01:52:33.061867 21840 leader_election.cc:302] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 3ee5f2a8853c40f88bc2a2a1ed479323; no voters:
I20240502 01:52:33.062145 21978 raft_consensus.cc:2781] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.062671 21978 raft_consensus.cc:686] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [term 1 LEADER]: Becoming Leader. State: Replica: 22defba24bbd4eacb7349f0158b38522, State: Running, Role: LEADER
I20240502 01:52:33.063198 21978 consensus_queue.cc:237] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.067627 21497 catalog_manager.cc:5484] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22defba24bbd4eacb7349f0158b38522 (127.20.243.3). New cstate: current_term: 1 leader_uuid: "22defba24bbd4eacb7349f0158b38522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:33.102622 21954 tablet.cc:2369] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:33.131187 21821 tablet.cc:2369] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:33.163576 21688 tablet.cc:2369] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:33.176324 21977 raft_consensus.cc:482] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.176527 21977 raft_consensus.cc:504] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.178248 21977 leader_election.cc:288] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255), 22defba24bbd4eacb7349f0158b38522 (127.20.243.3:41963)
I20240502 01:52:33.187026 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c17d10e240b44bfbfc539fb22cdfda2" candidate_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6620bb8e459945eb8e042f7806df6cd5" is_pre_election: true
I20240502 01:52:33.187345 21774 raft_consensus.cc:2454] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee5f2a8853c40f88bc2a2a1ed479323 in term 0.
I20240502 01:52:33.187319 21906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c17d10e240b44bfbfc539fb22cdfda2" candidate_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22defba24bbd4eacb7349f0158b38522" is_pre_election: true
I20240502 01:52:33.187780 21906 raft_consensus.cc:2454] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee5f2a8853c40f88bc2a2a1ed479323 in term 0.
I20240502 01:52:33.187948 21573 leader_election.cc:302] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee5f2a8853c40f88bc2a2a1ed479323, 6620bb8e459945eb8e042f7806df6cd5; no voters:
I20240502 01:52:33.188416 21977 raft_consensus.cc:2781] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.188544 21977 raft_consensus.cc:482] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.188634 21977 raft_consensus.cc:3037] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.190725 21977 raft_consensus.cc:504] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.191418 21977 leader_election.cc:288] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [CANDIDATE]: Term 1 election: Requested vote from peers 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255), 22defba24bbd4eacb7349f0158b38522 (127.20.243.3:41963)
I20240502 01:52:33.191926 21906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c17d10e240b44bfbfc539fb22cdfda2" candidate_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22defba24bbd4eacb7349f0158b38522"
I20240502 01:52:33.191922 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c17d10e240b44bfbfc539fb22cdfda2" candidate_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6620bb8e459945eb8e042f7806df6cd5"
I20240502 01:52:33.192111 21906 raft_consensus.cc:3037] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.192137 21774 raft_consensus.cc:3037] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.192950 21979 raft_consensus.cc:482] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.193127 21979 raft_consensus.cc:504] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.194624 21906 raft_consensus.cc:2454] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee5f2a8853c40f88bc2a2a1ed479323 in term 1.
I20240502 01:52:33.194842 21774 raft_consensus.cc:2454] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee5f2a8853c40f88bc2a2a1ed479323 in term 1.
I20240502 01:52:33.195187 21979 leader_election.cc:288] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22defba24bbd4eacb7349f0158b38522 (127.20.243.3:41963), 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113)
I20240502 01:52:33.195410 21572 leader_election.cc:302] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 3ee5f2a8853c40f88bc2a2a1ed479323; no voters:
I20240502 01:52:33.195902 21977 raft_consensus.cc:2781] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.196812 21977 raft_consensus.cc:686] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee5f2a8853c40f88bc2a2a1ed479323, State: Running, Role: LEADER
I20240502 01:52:33.197433 21977 consensus_queue.cc:237] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } }
I20240502 01:52:33.202726 21497 catalog_manager.cc:5484] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1). New cstate: current_term: 1 leader_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.204164 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "6620bb8e459945eb8e042f7806df6cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" is_pre_election: true
I20240502 01:52:33.204380 21640 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6620bb8e459945eb8e042f7806df6cd5 in term 0.
I20240502 01:52:33.204811 21709 leader_election.cc:302] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee5f2a8853c40f88bc2a2a1ed479323, 6620bb8e459945eb8e042f7806df6cd5; no voters:
I20240502 01:52:33.205266 21979 raft_consensus.cc:2781] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.205413 21979 raft_consensus.cc:482] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.205504 21979 raft_consensus.cc:3037] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.208298 21979 raft_consensus.cc:504] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.208879 21979 leader_election.cc:288] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [CANDIDATE]: Term 1 election: Requested vote from peers 22defba24bbd4eacb7349f0158b38522 (127.20.243.3:41963), 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113)
I20240502 01:52:33.208878 21906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "6620bb8e459945eb8e042f7806df6cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22defba24bbd4eacb7349f0158b38522" is_pre_election: true
I20240502 01:52:33.209162 21906 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6620bb8e459945eb8e042f7806df6cd5 in term 0.
I20240502 01:52:33.209686 21905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "6620bb8e459945eb8e042f7806df6cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22defba24bbd4eacb7349f0158b38522"
I20240502 01:52:33.209926 21905 raft_consensus.cc:3037] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.210549 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "6620bb8e459945eb8e042f7806df6cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
I20240502 01:52:33.210760 21640 raft_consensus.cc:3037] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.212035 21905 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6620bb8e459945eb8e042f7806df6cd5 in term 1.
I20240502 01:52:33.212746 21640 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6620bb8e459945eb8e042f7806df6cd5 in term 1.
I20240502 01:52:33.212421 21708 leader_election.cc:302] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 6620bb8e459945eb8e042f7806df6cd5; no voters:
I20240502 01:52:33.213239 21979 raft_consensus.cc:2781] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.213718 21979 raft_consensus.cc:686] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 6620bb8e459945eb8e042f7806df6cd5, State: Running, Role: LEADER
I20240502 01:52:33.214257 21979 consensus_queue.cc:237] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:52:33.219152 21497 catalog_manager.cc:5484] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2). New cstate: current_term: 1 leader_uuid: "6620bb8e459945eb8e042f7806df6cd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.225809 21452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:52:33.228634 21452 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 3ee5f2a8853c40f88bc2a2a1ed479323 to finish bootstrapping
I20240502 01:52:33.238921 21452 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 6620bb8e459945eb8e042f7806df6cd5 to finish bootstrapping
I20240502 01:52:33.247021 21452 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 22defba24bbd4eacb7349f0158b38522 to finish bootstrapping
I20240502 01:52:33.255993 21998 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.20.243.3:40189/maintenance-manager
http://127.20.243.3:40189/scans
http://127.20.243.1:36191/stacks
http://127.20.243.62:36437/startup
http://127.20.243.2:33909/maintenance-manager
http://127.20.243.62:36437/metrics
http://127.20.243.1:36191/threadz
http://127.20.243.3:40189/tablets
http://127.20.243.62:36437/ipki-ca-cert
http://127.20.243.2:33909/tablets
http://127.20.243.3:40189/metrics
http://127.20.243.62:36437/stacks
http://127.20.243.62:36437/tables
http://127.20.243.1:36191/metrics
http://127.20.243.62:36437/tablet-servers
http://127.20.243.1:36191/scans
http://127.20.243.1:36191/mem-trackers
http://127.20.243.62:36437/mem-trackers
http://127.20.243.3:40189/stacks
http://127.20.243.2:33909/mem-trackers
http://127.20.243.1:36191/startup
http://127.20.243.62:36437/threadz
http://127.20.243.2:33909/scans
http://127.20.243.1:36191/tablets
http://127.20.243.2:33909/transactions?tablet_id=4c17d10e240b44bfbfc539fb22cdfda2
http://127.20.243.1:36191/transactions?tablet_id=4c17d10e240b44bfbfc539fb22cdfda2
http://127.20.243.62:36437/threadz?group=all
http://127.20.243.62:36437/masters
http://127.20.243.62:36437/dump-entities
http://127.20.243.1:36191/threadz?group=all
http://127.20.243.2:33909/stacks
http://127.20.243.3:40189/transactions?tablet_id=4c17d10e240b44bfbfc539fb22cdfda2
http://127.20.243.3:40189/mem-trackers
http://127.20.243.2:33909/startup
http://127.20.243.2:33909/metrics
http://127.20.243.1:36191/maintenance-manager
http://127.20.243.2:33909/threadz
http://127.20.243.2:33909/threadz?group=all
http://127.20.243.3:40189/startup
http://127.20.243.3:40189/threadz?group=all
http://127.20.243.62:36437/ipki-ca-cert-pem
http://127.20.243.3:40189/threadz
http://127.20.243.62:36437/ipki-ca-cert-der
I20240502 01:52:33.260875 21452 linked_list-test-util.h:401] Running for: 30.000s
I20240502 01:52:33.261078 21452 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 01:52:33.261188 21452 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 01:52:33.261801 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:33.312330 21774 raft_consensus.cc:1261] T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Refusing update from remote peer 22defba24bbd4eacb7349f0158b38522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.315021 21640 raft_consensus.cc:1261] T b448622701684a588baf8dde2bf57cb9 P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Refusing update from remote peer 22defba24bbd4eacb7349f0158b38522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.315519 21905 raft_consensus.cc:1261] T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Refusing update from remote peer 3ee5f2a8853c40f88bc2a2a1ed479323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.316059 21906 raft_consensus.cc:1261] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Refusing update from remote peer 6620bb8e459945eb8e042f7806df6cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.316702 21979 consensus_queue.cc:1035] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.316974 21978 consensus_queue.cc:1035] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.317155 21993 consensus_queue.cc:1035] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.317724 21978 consensus_queue.cc:1035] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.323144 21640 raft_consensus.cc:1261] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Refusing update from remote peer 6620bb8e459945eb8e042f7806df6cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.324156 21995 consensus_queue.cc:1035] T 039bd9290f5e44169be2d8f53a9c87bc P 6620bb8e459945eb8e042f7806df6cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.324826 21774 raft_consensus.cc:1261] T 4c17d10e240b44bfbfc539fb22cdfda2 P 6620bb8e459945eb8e042f7806df6cd5 [term 1 FOLLOWER]: Refusing update from remote peer 3ee5f2a8853c40f88bc2a2a1ed479323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.325980 21993 consensus_queue.cc:1035] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:33.347807 22007 mvcc.cc:205] Tried to move back new op lower bound from 7023062029571469312 to 7023062029170208768. Current Snapshot: MvccSnapshot[applied={T|T < 7023062029571469312}]
I20240502 01:52:33.360987 22012 mvcc.cc:205] Tried to move back new op lower bound from 7023062029571469312 to 7023062029170208768. Current Snapshot: MvccSnapshot[applied={T|T < 7023062029571469312 or (T in {7023062029571469312})}]
I20240502 01:52:33.366842 22006 mvcc.cc:205] Tried to move back new op lower bound from 7023062029571469312 to 7023062029170208768. Current Snapshot: MvccSnapshot[applied={T|T < 7023062029571469312}]
I20240502 01:52:34.952862 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:35.791141 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:36.903235 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:38.528506 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:38.901132 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:38.938391 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:39.118458 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:39.807497 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:41.264962 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 2.321s user 0.599s sys 0.124s Metrics: {"bytes_written":2735,"cfile_init":1,"compiler_manager_pool.queue_time_us":21018,"delete_count":0,"dirs.queue_time_us":127162,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":74339,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2318,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7701,"thread_start_us":149169,"threads_started":2,"update_count":398}
I20240502 01:52:41.266789 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 133466 bytes on disk
I20240502 01:52:41.267756 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 01:52:41.268574 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=0.022372
I20240502 01:52:41.463717 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 2.344s user 0.527s sys 0.120s Metrics: {"bytes_written":3989,"cfile_init":1,"compiler_manager_pool.queue_time_us":9273,"delete_count":0,"dirs.queue_time_us":6479,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":93124,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3029,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7896,"thread_start_us":37955,"threads_started":2,"update_count":593}
I20240502 01:52:41.465838 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=0.031834
I20240502 01:52:41.508355 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.240s user 0.092s sys 0.020s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3644,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":39971,"lbm_read_time_us":480,"lbm_reads_lt_1ms":20,"lbm_write_time_us":337,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":39910,"threads_started":2,"update_count":398}
I20240502 01:52:41.689966 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 2.788s user 0.578s sys 0.078s Metrics: {"bytes_written":2525,"cfile_init":1,"compiler_manager_pool.queue_time_us":142454,"compiler_manager_pool.run_cpu_time_us":717127,"compiler_manager_pool.run_wall_time_us":2405164,"delete_count":0,"dirs.queue_time_us":9433,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":55652,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2200,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7668,"thread_start_us":39804,"threads_started":2,"update_count":365}
I20240502 01:52:41.692023 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 133145 bytes on disk
I20240502 01:52:41.693127 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
I20240502 01:52:41.693908 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=0.020744
I20240502 01:52:41.695796 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.230s user 0.066s sys 0.029s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5057,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":20818,"lbm_read_time_us":522,"lbm_reads_lt_1ms":24,"lbm_write_time_us":392,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":21078,"threads_started":2,"update_count":593}
I20240502 01:52:41.697010 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 135469 bytes on disk
I20240502 01:52:41.698029 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 01:52:41.960851 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.266s user 0.062s sys 0.047s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3388,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":83654,"lbm_read_time_us":542,"lbm_reads_lt_1ms":20,"lbm_write_time_us":351,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":86387,"threads_started":2,"update_count":365}
I20240502 01:52:42.188855 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:52:43.012954 22008 log.cc:924] Time spent T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20240502 01:52:43.013511 21988 log.cc:924] Time spent T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20240502 01:52:43.192160 22008 log.cc:924] Time spent T 4c17d10e240b44bfbfc539fb22cdfda2 P 22defba24bbd4eacb7349f0158b38522: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20240502 01:52:43.192615 22010 log.cc:924] Time spent T b448622701684a588baf8dde2bf57cb9 P 6620bb8e459945eb8e042f7806df6cd5: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240502 01:52:43.260845 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062070294192128 Inserted count: 18000
I20240502 01:52:43.785310 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:44.696342 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:45.723579 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:45.765619 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:45.772289 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:46.216178 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:46.510737 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.787s user 0.446s sys 0.113s Metrics: {"bytes_written":7697,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9162,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2181,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7860,"thread_start_us":9310,"threads_started":1,"update_count":1176}
I20240502 01:52:46.512706 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014694
I20240502 01:52:47.184723 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.410s user 0.466s sys 0.084s Metrics: {"bytes_written":8127,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6889,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":3042,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2511,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7894,"thread_start_us":6783,"threads_started":1,"update_count":1243}
I20240502 01:52:47.186911 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 129998 bytes on disk
I20240502 01:52:47.195688 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 01:52:47.197273 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014694
I20240502 01:52:47.560573 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.048s user 0.633s sys 0.028s Metrics: {"bytes_written":287425,"cfile_cache_hit":13,"cfile_cache_hit_bytes":45849,"cfile_cache_miss":63,"cfile_cache_miss_bytes":995391,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":8382,"dirs.run_cpu_time_us":898,"dirs.run_wall_time_us":10434,"drs_written":1,"lbm_read_time_us":3849,"lbm_reads_lt_1ms":103,"lbm_write_time_us":17833,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":48,"mutex_wait_us":150,"num_input_rowsets":2,"peak_mem_usage":509997,"rows_written":15528,"thread_start_us":29224,"threads_started":3}
I20240502 01:52:47.562232 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 283315 bytes on disk
I20240502 01:52:47.562922 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 01:52:47.635654 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.419s user 0.505s sys 0.072s Metrics: {"bytes_written":2283,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12183,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2298,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8095,"thread_start_us":12327,"threads_started":1,"update_count":327}
I20240502 01:52:47.637441 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 141081 bytes on disk
I20240502 01:52:47.638177 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 01:52:47.639271 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014686
I20240502 01:52:48.385434 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.186s user 0.642s sys 0.014s Metrics: {"bytes_written":288504,"cfile_cache_hit":15,"cfile_cache_hit_bytes":137794,"cfile_cache_miss":61,"cfile_cache_miss_bytes":907234,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":2356,"dirs.run_cpu_time_us":939,"dirs.run_wall_time_us":5313,"drs_written":1,"lbm_read_time_us":12893,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":96,"lbm_write_time_us":3319,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":512242,"rows_written":15595,"thread_start_us":14729,"threads_started":3}
I20240502 01:52:48.387035 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 284419 bytes on disk
I20240502 01:52:48.387984 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 01:52:48.489548 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.850s user 0.654s sys 0.014s Metrics: {"bytes_written":295205,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93453,"cfile_cache_miss":62,"cfile_cache_miss_bytes":980411,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":9080,"dirs.run_cpu_time_us":1201,"dirs.run_wall_time_us":18313,"drs_written":1,"lbm_read_time_us":2648,"lbm_reads_lt_1ms":98,"lbm_write_time_us":3172,"lbm_writes_lt_1ms":51,"mutex_wait_us":360,"num_input_rowsets":2,"peak_mem_usage":525524,"rows_written":15991,"spinlock_wait_cycles":1920,"thread_start_us":23991,"threads_started":3}
I20240502 01:52:48.490700 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 292148 bytes on disk
I20240502 01:52:48.491426 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 01:52:49.013962 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:49.922622 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:51.051007 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:52.324688 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:52.401819 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:52.507057 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:52:52.809484 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:53.284660 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062111162695680 Inserted count: 36650
I20240502 01:52:53.299393 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.792s user 0.499s sys 0.026s Metrics: {"bytes_written":142354,"cfile_init":1,"dirs.queue_time_us":7228,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":25798,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2068,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7750,"thread_start_us":7368,"threads_started":1}
I20240502 01:52:53.300849 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 135113 bytes on disk
I20240502 01:52:53.302083 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240502 01:52:53.303005 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014710
I20240502 01:52:53.308024 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.906s user 0.445s sys 0.059s Metrics: {"bytes_written":145728,"cfile_init":1,"dirs.queue_time_us":31729,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":944,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2059,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7984,"spinlock_wait_cycles":223104,"thread_start_us":31589,"threads_started":1}
I20240502 01:52:53.309418 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014710
I20240502 01:52:53.784529 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.460s user 0.518s sys 0.043s Metrics: {"bytes_written":1053,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":65542,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":3314,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2962,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8051,"spinlock_wait_cycles":12389120,"thread_start_us":65397,"threads_started":1,"update_count":137}
I20240502 01:52:53.785923 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014710
I20240502 01:52:54.553282 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:55.683220 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 2.379s user 1.116s sys 0.061s Metrics: {"bytes_written":2118,"cfile_cache_hit":11,"cfile_cache_hit_bytes":46407,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1557857,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":88301,"dirs.run_cpu_time_us":1397,"dirs.run_wall_time_us":22496,"drs_written":1,"lbm_read_time_us":19292,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":4643,"lbm_writes_lt_1ms":66,"mutex_wait_us":26,"num_input_rowsets":2,"peak_mem_usage":779787,"reinsert_count":0,"rows_written":23741,"spinlock_wait_cycles":7552,"thread_start_us":113187,"threads_started":4,"update_count":299}
I20240502 01:52:55.684600 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 436921 bytes on disk
I20240502 01:52:55.685201 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 01:52:55.693446 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=0.005608
I20240502 01:52:55.897284 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.204s user 0.113s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3165,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2424,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":691,"lbm_read_time_us":395,"lbm_reads_lt_1ms":20,"lbm_write_time_us":9325,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"mutex_wait_us":2329,"reinsert_count":0,"update_count":299}
I20240502 01:52:56.333580 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 3.021s user 0.996s sys 0.078s Metrics: {"bytes_written":1091,"cfile_cache_hit":13,"cfile_cache_hit_bytes":227714,"cfile_cache_miss":71,"cfile_cache_miss_bytes":1365668,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":31100,"dirs.run_cpu_time_us":1222,"dirs.run_wall_time_us":11764,"drs_written":1,"lbm_read_time_us":4006,"lbm_reads_lt_1ms":103,"lbm_write_time_us":6457,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":65,"num_input_rowsets":2,"peak_mem_usage":777255,"reinsert_count":0,"rows_written":23579,"spinlock_wait_cycles":3328,"thread_start_us":154192,"threads_started":3,"update_count":137}
I20240502 01:52:56.334843 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=0.002910
I20240502 01:52:56.484330 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: MajorDeltaCompactionOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.149s user 0.089s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1606,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":83,"lbm_read_time_us":435,"lbm_reads_lt_1ms":24,"lbm_write_time_us":337,"lbm_writes_lt_1ms":8,"mutex_wait_us":2,"reinsert_count":0,"update_count":137}
I20240502 01:52:56.485159 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 435283 bytes on disk
I20240502 01:52:56.485929 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 01:52:56.561400 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:56.691593 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 2.902s user 0.992s sys 0.059s Metrics: {"bytes_written":431997,"cfile_cache_hit":13,"cfile_cache_hit_bytes":179652,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1415672,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":21943,"dirs.run_cpu_time_us":1619,"dirs.run_wall_time_us":36035,"drs_written":1,"lbm_read_time_us":4034,"lbm_reads_lt_1ms":111,"lbm_write_time_us":33353,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":60,"mutex_wait_us":77,"num_input_rowsets":2,"peak_mem_usage":779709,"rows_written":23579,"spinlock_wait_cycles":3456,"thread_start_us":43488,"threads_started":4}
I20240502 01:52:56.692720 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 436581 bytes on disk
I20240502 01:52:56.693504 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 01:52:58.655514 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(b448622701684a588baf8dde2bf57cb9): perf score=1.000000
I20240502 01:52:58.698805 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling FlushMRSOp(b448622701684a588baf8dde2bf57cb9): perf score=1.000000
I20240502 01:52:58.743430 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(b448622701684a588baf8dde2bf57cb9): perf score=1.000000
I20240502 01:52:58.854669 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:52:59.320614 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 0.659s user 0.455s sys 0.139s Metrics: {"bytes_written":141617,"cfile_init":1,"dirs.queue_time_us":157511,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":2670,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2188,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7615,"thread_start_us":157399,"threads_started":1}
I20240502 01:52:59.331156 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9): 142235 bytes on disk
I20240502 01:52:59.331781 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 01:52:59.332585 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(4c17d10e240b44bfbfc539fb22cdfda2): perf score=1.000000
I20240502 01:52:59.943665 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: FlushMRSOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 1.245s user 0.466s sys 0.018s Metrics: {"bytes_written":477,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12614,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":5798,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2319,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7626,"thread_start_us":12729,"threads_started":1,"update_count":48}
I20240502 01:52:59.946067 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9): 142374 bytes on disk
I20240502 01:52:59.946934 21711 maintenance_manager.cc:643] P 6620bb8e459945eb8e042f7806df6cd5: UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 01:52:59.947615 21821 maintenance_manager.cc:419] P 6620bb8e459945eb8e042f7806df6cd5: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:53:00.475381 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 1.731s user 0.478s sys 0.055s Metrics: {"bytes_written":541,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":120360,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":33809,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4445,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7635,"thread_start_us":162563,"threads_started":1,"update_count":57}
I20240502 01:53:00.476917 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:53:00.551541 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(4c17d10e240b44bfbfc539fb22cdfda2) complete. Timing: real 1.219s user 0.411s sys 0.106s Metrics: {"bytes_written":1327,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15946,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1507,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2269,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7679,"thread_start_us":15227,"threads_started":1,"update_count":180}
I20240502 01:53:00.552865 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(4c17d10e240b44bfbfc539fb22cdfda2): 142321 bytes on disk
I20240502 01:53:00.556303 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(4c17d10e240b44bfbfc539fb22cdfda2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 01:53:00.557299 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.000000
I20240502 01:53:00.925904 22033 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 01:53:00.931738 22033 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f6704950dcd google::LogMessage::Fail() at ??:0
@ 0x7f6704954b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f67049507cc google::LogMessage::Flush() at ??:0
@ 0x7f6704951f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6707579c80 kudu::operator-() at ??:0
@ 0x7f67141fa14a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f67141c45de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f67141c4336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f67141712d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f6716c3ee47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f6716c4d9fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f6716c4d862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f671068302a std::function<>::operator()() at ??:0
@ 0x7f671067b7e6 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f67106767f1 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f6710674927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f67106b2089 handle_request at ??:0
@ 0x7f67106b476b worker_thread at ??:0
@ 0x7f670a9c56db start_thread at ??:0
@ 0x7f670380371f clone at ??:0
I20240502 01:53:01.229648 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.752s user 0.424s sys 0.015s Metrics: {"bytes_written":6343,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":455,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":953,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2070,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8465,"thread_start_us":320,"threads_started":1,"update_count":960}
I20240502 01:53:01.230677 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling LogGCOp(039bd9290f5e44169be2d8f53a9c87bc): free 1049407 bytes of WAL
I20240502 01:53:01.231024 21575 log_reader.cc:386] T 039bd9290f5e44169be2d8f53a9c87bc: removed 1 log segments from log reader
I20240502 01:53:01.231128 21575 log.cc:1076] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323: Deleting log segment in path: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/wals/039bd9290f5e44169be2d8f53a9c87bc/wal-000000001 (ops 1-739)
I20240502 01:53:01.231690 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: LogGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240502 01:53:01.232122 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 142860 bytes on disk
I20240502 01:53:01.232640 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 01:53:01.233248 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9): 142431 bytes on disk
I20240502 01:53:01.233680 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 01:53:01.234174 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014679
W20240502 01:53:01.343168 21958 meta_cache.cc:302] tablet 039bd9290f5e44169be2d8f53a9c87bc: replica 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255) has failed: Network error: recv got EOF from 127.20.243.2:35255 (error 108)
W20240502 01:53:01.346957 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.355801 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
I20240502 01:53:01.364835 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: FlushMRSOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.803s user 0.463s sys 0.040s Metrics: {"bytes_written":6539,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":430,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":945,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2256,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8335,"thread_start_us":266,"threads_started":1,"update_count":992}
I20240502 01:53:01.366055 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling LogGCOp(039bd9290f5e44169be2d8f53a9c87bc): free 1049508 bytes of WAL
I20240502 01:53:01.366447 21842 log_reader.cc:386] T 039bd9290f5e44169be2d8f53a9c87bc: removed 1 log segments from log reader
I20240502 01:53:01.366580 21842 log.cc:1076] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522: Deleting log segment in path: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal/wals/039bd9290f5e44169be2d8f53a9c87bc/wal-000000001 (ops 1-736)
I20240502 01:53:01.367264 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: LogGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240502 01:53:01.367826 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 140379 bytes on disk
I20240502 01:53:01.368449 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 01:53:01.369295 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc): perf score=1.014679
W20240502 01:53:01.388103 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.396456 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.420099 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
I20240502 01:53:01.429296 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:53:01.431740 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.471786 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.487788 21840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111)
W20240502 01:53:01.487975 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.512493 21840 consensus_peers.cc:485] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.526420 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.543107 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.574748 21573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111)
W20240502 01:53:01.583282 21573 consensus_peers.cc:485] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.584594 21865 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.587574 21865 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.592902 21597 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.608620 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.610812 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.615442 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.638335 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.647619 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.656158 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.677507 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.681715 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.693256 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.725328 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.743342 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.781961 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.793198 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.805472 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.822152 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.848388 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.872259 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.888862 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.922941 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.928260 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:01.953760 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.001003 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.015118 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.039013 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.044515 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.107491 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.117501 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.140377 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.159086 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.215127 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.244120 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.251657 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.289319 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.327939 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.367305 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
W20240502 01:53:02.380348 21598 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43420: Illegal state: replica 3ee5f2a8853c40f88bc2a2a1ed479323 is not leader of this config: current role FOLLOWER
I20240502 01:53:02.402362 22285 raft_consensus.cc:482] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6620bb8e459945eb8e042f7806df6cd5)
I20240502 01:53:02.402617 22285 raft_consensus.cc:504] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:53:02.403692 22285 leader_election.cc:288] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113), 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255)
I20240502 01:53:02.405685 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" is_pre_election: true
I20240502 01:53:02.406002 21640 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 1.
I20240502 01:53:02.407200 21840 leader_election.cc:302] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 3ee5f2a8853c40f88bc2a2a1ed479323; no voters:
I20240502 01:53:02.412416 22285 raft_consensus.cc:2781] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:53:02.412573 22285 raft_consensus.cc:482] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6620bb8e459945eb8e042f7806df6cd5)
I20240502 01:53:02.412696 22285 raft_consensus.cc:3037] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:02.416786 22285 raft_consensus.cc:504] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:53:02.417680 22285 leader_election.cc:288] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 election: Requested vote from peers 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113), 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255)
W20240502 01:53:02.418154 21840 leader_election.cc:334] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111)
I20240502 01:53:02.420097 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039bd9290f5e44169be2d8f53a9c87bc" candidate_uuid: "22defba24bbd4eacb7349f0158b38522" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323"
I20240502 01:53:02.420403 21640 raft_consensus.cc:3037] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:02.424003 21640 raft_consensus.cc:2454] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22defba24bbd4eacb7349f0158b38522 in term 2.
W20240502 01:53:02.424667 21958 meta_cache.cc:302] tablet 039bd9290f5e44169be2d8f53a9c87bc: replica 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255) has failed: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111)
W20240502 01:53:02.426630 21866 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46686: Illegal state: replica 22defba24bbd4eacb7349f0158b38522 is not leader of this config: current role FOLLOWER
I20240502 01:53:02.427994 21840 leader_election.cc:302] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22defba24bbd4eacb7349f0158b38522, 3ee5f2a8853c40f88bc2a2a1ed479323; no voters:
W20240502 01:53:02.429497 21840 leader_election.cc:334] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111)
I20240502 01:53:02.430271 22285 raft_consensus.cc:2781] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:02.430992 22285 raft_consensus.cc:686] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [term 2 LEADER]: Becoming Leader. State: Replica: 22defba24bbd4eacb7349f0158b38522, State: Running, Role: LEADER
I20240502 01:53:02.431453 22285 consensus_queue.cc:237] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 999, Committed index: 999, Last appended: 1.999, Last appended by leader: 999, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } }
I20240502 01:53:02.441336 21494 catalog_manager.cc:5484] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 reported cstate change: term changed from 1 to 2, leader changed from 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2) to 22defba24bbd4eacb7349f0158b38522 (127.20.243.3). New cstate: current_term: 2 leader_uuid: "22defba24bbd4eacb7349f0158b38522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22defba24bbd4eacb7349f0158b38522" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 41963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6620bb8e459945eb8e042f7806df6cd5" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 35255 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:02.481439 21840 consensus_peers.cc:485] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:02.482564 21640 raft_consensus.cc:1261] T 039bd9290f5e44169be2d8f53a9c87bc P 3ee5f2a8853c40f88bc2a2a1ed479323 [term 2 FOLLOWER]: Refusing update from remote peer 22defba24bbd4eacb7349f0158b38522: Log matching property violated. Preceding OpId in replica: term: 1 index: 999. Preceding OpId from leader: term: 2 index: 1001. (index mismatch)
I20240502 01:53:02.483196 22285 consensus_queue.cc:1035] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee5f2a8853c40f88bc2a2a1ed479323" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1000, Last known committed idx: 999, Time since last communication: 0.000s
I20240502 01:53:02.738305 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.504s user 1.343s sys 0.059s Metrics: {"bytes_written":584744,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1020,"cfile_cache_miss":96,"cfile_cache_miss_bytes":2185410,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":11041,"dirs.run_cpu_time_us":2255,"dirs.run_wall_time_us":22481,"drs_written":1,"lbm_read_time_us":7113,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":131,"lbm_write_time_us":7111,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":1063247,"rows_written":32044,"thread_start_us":15763,"threads_started":5}
I20240502 01:53:02.740267 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 599209 bytes on disk
I20240502 01:53:02.741655 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 01:53:02.742712 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling FlushMRSOp(4c17d10e240b44bfbfc539fb22cdfda2): perf score=1.000000
I20240502 01:53:02.765466 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: CompactRowSetsOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 1.396s user 1.318s sys 0.042s Metrics: {"bytes_written":585167,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1020,"cfile_cache_miss":96,"cfile_cache_miss_bytes":2186578,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":48821,"dirs.run_cpu_time_us":2241,"dirs.run_wall_time_us":37038,"drs_written":1,"lbm_read_time_us":5922,"lbm_reads_lt_1ms":132,"lbm_write_time_us":6487,"lbm_writes_lt_1ms":76,"mutex_wait_us":400,"num_input_rowsets":2,"peak_mem_usage":1064314,"rows_written":32076,"thread_start_us":61348,"threads_started":5}
I20240502 01:53:02.766985 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling MajorDeltaCompactionOp(4c17d10e240b44bfbfc539fb22cdfda2): perf score=0.010837
I20240502 01:53:02.807926 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: MajorDeltaCompactionOp(4c17d10e240b44bfbfc539fb22cdfda2) complete. Timing: real 0.040s user 0.038s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1841,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":427,"lbm_reads_lt_1ms":20,"lbm_write_time_us":345,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":180}
I20240502 01:53:02.809049 21954 maintenance_manager.cc:419] P 22defba24bbd4eacb7349f0158b38522: Scheduling UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc): 599919 bytes on disk
I20240502 01:53:02.809725 21842 maintenance_manager.cc:643] P 22defba24bbd4eacb7349f0158b38522: UndoDeltaBlockGCOp(039bd9290f5e44169be2d8f53a9c87bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 01:53:03.108474 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:03.267478 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062152213856256 Inserted count: 50100
I20240502 01:53:03.267642 21452 linked_list-test-util.h:421] Finished inserting list. Added 50100 in chain
I20240502 01:53:03.267709 21452 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 01:53:03.267793 21452 linked_list-test-util.h:424] 0: -7435053058340814848
I20240502 01:53:03.267859 21452 linked_list-test-util.h:424] 1: -7456550844189900799
I20240502 01:53:03.267932 21452 linked_list-test-util.h:424] 2: -7477626417573986302
I20240502 01:53:03.268002 21452 linked_list-test-util.h:424] 3: -7498701990958071805
I20240502 01:53:03.268075 21452 linked_list-test-util.h:424] 4: 3491772713538289668
I20240502 01:53:03.268128 21452 linked_list-test-util.h:424] 5: 3470415665177493509
I20240502 01:53:03.268193 21452 linked_list-test-util.h:424] 6: 3449199354305118214
I20240502 01:53:03.268250 21452 linked_list-test-util.h:424] 7: 3428123780921032711
I20240502 01:53:03.268321 21452 linked_list-test-util.h:424] 8: -4028145588292157432
I20240502 01:53:03.268378 21452 linked_list-test-util.h:424] 9: -4049502636652953591
I20240502 01:53:03.268445 21452 linked_list-test-util.h:424] 10: -4070578210037039094
I20240502 01:53:03.268496 21452 linked_list-test-util.h:424] 11: -4091794520909414389
I20240502 01:53:03.268561 21452 linked_list-test-util.h:424] 12: 6898680183586947084
I20240502 01:53:03.268613 21452 linked_list-test-util.h:424] 13: 6877323135226150925
I20240502 01:53:03.268682 21452 linked_list-test-util.h:424] 14: 6856247561842065422
I20240502 01:53:03.268740 21452 linked_list-test-util.h:424] 15: 6835171988457979919
I20240502 01:53:03.268811 21452 linked_list-test-util.h:424] 16: -621097380755210224
I20240502 01:53:03.268863 21452 linked_list-test-util.h:424] 17: -642595166604296175
I20240502 01:53:03.268929 21452 linked_list-test-util.h:424] 18: -663670739988381678
I20240502 01:53:03.268980 21452 linked_list-test-util.h:424] 19: -684746313372467181
I20240502 01:53:03.269044 21452 linked_list-test-util.h:424] 20: -8141015682585657324
I20240502 01:53:03.269109 21452 linked_list-test-util.h:424] 21: -8162372730946453483
I20240502 01:53:03.269186 21452 linked_list-test-util.h:424] 22: -8183589041818828778
I20240502 01:53:03.269240 21452 linked_list-test-util.h:424] 23: -8204664615202914281
I20240502 01:53:03.269309 21452 linked_list-test-util.h:424] 24: 2785810089293447192
I20240502 01:53:03.269361 21452 linked_list-test-util.h:424] 25: 2764453040932651033
I20240502 01:53:03.269425 21452 linked_list-test-util.h:424] 26: 2743377467548565530
I20240502 01:53:03.269476 21452 linked_list-test-util.h:424] 27: -4712891901664624613
I20240502 01:53:03.269541 21452 linked_list-test-util.h:424] 28: -4734108212536999908
I20240502 01:53:03.269593 21452 linked_list-test-util.h:424] 29: -4755465260897796067
I20240502 01:53:03.269657 21452 linked_list-test-util.h:424] 30: -4776540834281881570
I20240502 01:53:03.269707 21452 linked_list-test-util.h:424] 31: 6213933870214479903
I20240502 01:53:03.269778 21452 linked_list-test-util.h:424] 32: 6192858296830394400
I20240502 01:53:03.269829 21452 linked_list-test-util.h:424] 33: 6171360510981308449
I20240502 01:53:03.269898 21452 linked_list-test-util.h:424] 34: 6150284937597222946
I20240502 01:53:03.269949 21452 linked_list-test-util.h:424] 35: -1305984431615967197
I20240502 01:53:03.270015 21452 linked_list-test-util.h:424] 36: -1327060005000052700
I20240502 01:53:03.270066 21452 linked_list-test-util.h:424] 37: -1348417053360848859
I20240502 01:53:03.270139 21452 linked_list-test-util.h:424] 38: -1369633364233224154
I20240502 01:53:03.270190 21452 linked_list-test-util.h:424] 39: -8825902733446414297
I20240502 01:53:03.270258 21452 linked_list-test-util.h:424] 40: -8846978306830499800
I20240502 01:53:03.270334 21452 linked_list-test-util.h:424] 41: -8868335355191295959
I20240502 01:53:03.270401 21452 linked_list-test-util.h:424] 42: -8889410928575381462
I20240502 01:53:03.270453 21452 linked_list-test-util.h:424] 43: 2101063775920980011
I20240502 01:53:03.270519 21452 linked_list-test-util.h:424] 44: 2079847465048604716
I20240502 01:53:03.270565 21452 linked_list-test-util.h:424] 45: 2058490416687808557
I20240502 01:53:03.270625 21452 linked_list-test-util.h:424] 46: 2037414843303723054
I20240502 01:53:03.270680 21452 linked_list-test-util.h:424] 47: -5418854525909467089
I20240502 01:53:03.270741 21452 linked_list-test-util.h:424] 48: -5439930099293552592
I20240502 01:53:03.270793 21452 linked_list-test-util.h:424] 49: -5461427885142638543
I20240502 01:53:03.359575 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614763255418 usec, L: 1 Expected Rows: 18000
I20240502 01:53:03.372471 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: FlushMRSOp(4c17d10e240b44bfbfc539fb22cdfda2) complete. Timing: real 0.629s user 0.511s sys 0.047s Metrics: {"bytes_written":149224,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":952,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2324,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8106,"spinlock_wait_cycles":896}
I20240502 01:53:03.373502 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling MajorDeltaCompactionOp(b448622701684a588baf8dde2bf57cb9): perf score=0.004455
I20240502 01:53:03.379746 21452 linked_list-test-util.h:528] Calling callback on tserver 22defba24bbd4eacb7349f0158b38522
I20240502 01:53:03.417450 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: MajorDeltaCompactionOp(b448622701684a588baf8dde2bf57cb9) complete. Timing: real 0.044s user 0.040s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":714,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":474,"lbm_read_time_us":454,"lbm_reads_lt_1ms":20,"lbm_write_time_us":385,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":276,"threads_started":1,"update_count":57}
I20240502 01:53:03.418401 21688 maintenance_manager.cc:419] P 3ee5f2a8853c40f88bc2a2a1ed479323: Scheduling UndoDeltaBlockGCOp(4c17d10e240b44bfbfc539fb22cdfda2): 151659 bytes on disk
I20240502 01:53:03.419090 21575 maintenance_manager.cc:643] P 3ee5f2a8853c40f88bc2a2a1ed479323: UndoDeltaBlockGCOp(4c17d10e240b44bfbfc539fb22cdfda2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240502 01:53:03.653497 21840 consensus_peers.cc:485] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:53:03.993772 21573 consensus_peers.cc:485] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:53:04.068900 21452 linked_list-test-util.h:767] Done collecting results (18000 rows in 689.323ms)
I20240502 01:53:04.081684 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:04.107653 21452 linked_list-test-util.h:556] Snapshot: P: 1714614763255418 usec, L: 1 verified. Result: OK
I20240502 01:53:04.112324 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614773233080 usec, L: 1 Expected Rows: 36650
I20240502 01:53:04.129360 21452 linked_list-test-util.h:528] Calling callback on tserver 22defba24bbd4eacb7349f0158b38522
I20240502 01:53:04.796823 21452 linked_list-test-util.h:767] Done collecting results (36650 rows in 667.609ms)
I20240502 01:53:04.856343 21452 linked_list-test-util.h:556] Snapshot: P: 1714614773233080 usec, L: 1 verified. Result: OK
I20240502 01:53:04.860021 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614783255336 usec, L: 1 Expected Rows: 50100
I20240502 01:53:04.869145 21452 linked_list-test-util.h:528] Calling callback on tserver 22defba24bbd4eacb7349f0158b38522
W20240502 01:53:04.984611 21840 consensus_peers.cc:485] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:53:05.377118 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:53:05.978271 21840 consensus_peers.cc:485] T b448622701684a588baf8dde2bf57cb9 P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:53:06.042908 21999 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:06.071085 21452 linked_list-test-util.h:767] Done collecting results (50100 rows in 1202.07ms)
I20240502 01:53:06.147603 21452 linked_list-test-util.h:556] Snapshot: P: 1714614783255336 usec, L: 1 verified. Result: OK
I20240502 01:53:06.150616 21452 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 50100
W20240502 01:53:06.441408 21573 consensus_peers.cc:485] T 4c17d10e240b44bfbfc539fb22cdfda2 P 3ee5f2a8853c40f88bc2a2a1ed479323 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:53:06.834764 21840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 01:53:07.005728 21573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 01:53:07.267061 22001 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 01:53:07.324719 21840 consensus_peers.cc:485] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 -> Peer 6620bb8e459945eb8e042f7806df6cd5 (127.20.243.2:35255): Couldn't send request to peer 6620bb8e459945eb8e042f7806df6cd5. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:53:07.390578 21452 linked_list-test-util.h:767] Done collecting results (50100 rows in 1230.53ms)
I20240502 01:53:07.449123 21452 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240502 01:53:07.449644 21452 linked_list-test-util.h:692] Successfully verified 50100 rows
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs:
TabletServer: 6620bb8e459945eb8e042f7806df6cd5, Rpc address: host: "127.20.243.2" port: 35255 (Network error: Client connection negotiation failed: client connection to 127.20.243.2:35255: connect: Connection refused (error 111))
I20240502 01:53:07.467375 21452 webui_checker.cc:88] shutting down CURL thread
I20240502 01:53:07.936379 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 21556
I20240502 01:53:07.938026 21682 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.164274 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 21556
I20240502 01:53:08.222708 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 21823
W20240502 01:53:08.223538 21840 consensus_peers.cc:485] T 039bd9290f5e44169be2d8f53a9c87bc P 22defba24bbd4eacb7349f0158b38522 -> Peer 3ee5f2a8853c40f88bc2a2a1ed479323 (127.20.243.1:37113): Couldn't send request to peer 3ee5f2a8853c40f88bc2a2a1ed479323. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:37113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:08.223778 21948 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.466131 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 21823
I20240502 01:53:08.516878 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 21465
I20240502 01:53:08.518081 21527 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.639035 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 21465
2024-05-02T01:53:08Z chronyd exiting
I20240502 01:53:08.679356 21452 test_util.cc:175] -----------------------------------------------
I20240502 01:53:08.679463 21452 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714614750802054-21452-0
[ FAILED ] LinkedListTest.TestLoadAndVerify (37810 ms)
[ RUN ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240502 01:53:08.681258 21452 test_util.cc:268] Using random seed: -580391401
I20240502 01:53:08.682940 21452 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 01:53:08.682998 21452 linked_list-test.cc:101] --------------
I20240502 01:53:08.683041 21452 linked_list-test.cc:102] 50 chains
I20240502 01:53:08.683084 21452 linked_list-test.cc:103] 3 tablets
I20240502 01:53:08.683128 21452 linked_list-test.cc:104] Mutations on
I20240502 01:53:08.683163 21452 linked_list-test.cc:105] --------------
I20240502 01:53:08.683291 21452 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:08.683332 21452 ts_itest-base.cc:113] --------------
I20240502 01:53:08.683378 21452 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:53:08.683413 21452 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:08.683462 21452 ts_itest-base.cc:116] --------------
2024-05-02T01:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:08Z Disabled control of system clock
I20240502 01:53:08.715889 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.243.62:41431
--webserver_interface=127.20.243.62
--webserver_port=0
--builtin_ntp_servers=127.20.243.20:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.243.62:41431
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 01:53:09.064596 22366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.064918 22366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.065078 22366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.072505 22366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:09.072616 22366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.072695 22366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:09.072780 22366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:09.081857 22366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.243.62:41431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.243.62:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.20.243.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.083294 22366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.085072 22366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.096295 22371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.096786 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.097417 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.097986 22366 server_base.cc:1034] running on GCE node
I20240502 01:53:09.098824 22366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.100235 22366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.101454 22366 hybrid_clock.cc:648] HybridClock initialized: now 1714614789101394 us; error 44 us; skew 500 ppm
I20240502 01:53:09.101774 22366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.104123 22366 webserver.cc:460] Webserver started at http://127.20.243.62:43603/ using document root <none> and password file <none>
I20240502 01:53:09.104878 22366 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.105015 22366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.105324 22366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.108107 22366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/instance:
uuid: "d18b50aa175d482096bda7cecd9f5082"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.108865 22366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal/instance:
uuid: "d18b50aa175d482096bda7cecd9f5082"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.114176 22366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 01:53:09.117637 22381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.119037 22366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:09.119211 22366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
uuid: "d18b50aa175d482096bda7cecd9f5082"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.119402 22366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.139292 22366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:09.140192 22366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.140491 22366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.159467 22366 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.62:41431
I20240502 01:53:09.159500 22433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.62:41431 every 8 connection(s)
I20240502 01:53:09.160888 22366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/data/info.pb
I20240502 01:53:09.164808 22434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:09.168174 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 22366
I20240502 01:53:09.168404 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/master-0/wal/instance
I20240502 01:53:09.175770 22434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082: Bootstrap starting.
I20240502 01:53:09.179405 22434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:09.180481 22434 log.cc:824] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:09.183144 22434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082: No bootstrap required, opened a new log
I20240502 01:53:09.189118 22434 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } }
I20240502 01:53:09.189478 22434 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:09.189579 22434 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18b50aa175d482096bda7cecd9f5082, State: Initialized, Role: FOLLOWER
I20240502 01:53:09.190161 22434 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } }
I20240502 01:53:09.190385 22434 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:09.190513 22434 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:09.190665 22434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:09.192890 22434 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } }
I20240502 01:53:09.193392 22434 leader_election.cc:302] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18b50aa175d482096bda7cecd9f5082; no voters:
I20240502 01:53:09.194007 22434 leader_election.cc:288] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:09.194159 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:09.195312 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [term 1 LEADER]: Becoming Leader. State: Replica: d18b50aa175d482096bda7cecd9f5082, State: Running, Role: LEADER
I20240502 01:53:09.195956 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } }
I20240502 01:53:09.196296 22434 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:09.198719 22441 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18b50aa175d482096bda7cecd9f5082. Latest consensus state: current_term: 1 leader_uuid: "d18b50aa175d482096bda7cecd9f5082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } } }
I20240502 01:53:09.198799 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d18b50aa175d482096bda7cecd9f5082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18b50aa175d482096bda7cecd9f5082" member_type: VOTER last_known_addr { host: "127.20.243.62" port: 41431 } } }
I20240502 01:53:09.199112 22441 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.199211 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.200217 22447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:09.207129 22447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:09.207834 22456 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 01:53:09.216080 22447 catalog_manager.cc:1347] Generated new cluster ID: 0fde7224b68e4acfb3436a17a9cb8417
I20240502 01:53:09.216225 22447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:09.246052 22447 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:09.247957 22447 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:09.261109 22447 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d18b50aa175d482096bda7cecd9f5082: Generated new TSK 0
I20240502 01:53:09.262347 22447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:09.273422 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.1:0
--local_ip_for_outbound_sockets=127.20.243.1
--webserver_interface=127.20.243.1
--webserver_port=0
--tserver_master_addrs=127.20.243.62:41431
--builtin_ntp_servers=127.20.243.20:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 01:53:09.614557 22458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.614883 22458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.615047 22458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.622736 22458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.622972 22458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.1
I20240502 01:53:09.632153 22458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.243.1
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.243.62:41431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.1
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.633510 22458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.635246 22458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.647434 22464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.647987 22465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.648404 22467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.649124 22458 server_base.cc:1034] running on GCE node
I20240502 01:53:09.649696 22458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.650658 22458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.651897 22458 hybrid_clock.cc:648] HybridClock initialized: now 1714614789651830 us; error 36 us; skew 500 ppm
I20240502 01:53:09.652176 22458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.654481 22458 webserver.cc:460] Webserver started at http://127.20.243.1:39805/ using document root <none> and password file <none>
I20240502 01:53:09.655187 22458 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.655297 22458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.655555 22458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.658207 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/instance:
uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.658968 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/instance:
uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.663982 22458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:09.667248 22473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.668532 22458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:09.668679 22458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:09.668848 22458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.705891 22458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:09.706847 22458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.707131 22458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.708107 22458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:09.709734 22458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:09.709810 22458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.709913 22458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:09.709959 22458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.742270 22458 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.1:34495
I20240502 01:53:09.742341 22589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.1:34495 every 8 connection(s)
I20240502 01:53:09.743664 22458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
I20240502 01:53:09.748989 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 22458
I20240502 01:53:09.749244 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/instance
I20240502 01:53:09.752874 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.2:0
--local_ip_for_outbound_sockets=127.20.243.2
--webserver_interface=127.20.243.2
--webserver_port=0
--tserver_master_addrs=127.20.243.62:41431
--builtin_ntp_servers=127.20.243.20:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 01:53:09.758157 22590 heartbeater.cc:347] Connected to a master server at 127.20.243.62:41431
I20240502 01:53:09.758505 22590 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:09.759380 22590 heartbeater.cc:511] Master 127.20.243.62:41431 requested a full tablet report, sending...
I20240502 01:53:09.761663 22398 ts_manager.cc:194] Registered new tserver with Master: c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495)
I20240502 01:53:09.763360 22398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.1:54755
W20240502 01:53:10.095279 22594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.095613 22594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.095772 22594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.103611 22594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.103844 22594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.2
I20240502 01:53:10.113170 22594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.20.243.2
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.243.62:41431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.2
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.114658 22594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.116426 22594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.129320 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.129976 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.130545 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.131278 22594 server_base.cc:1034] running on GCE node
I20240502 01:53:10.131891 22594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.132823 22594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.134040 22594 hybrid_clock.cc:648] HybridClock initialized: now 1714614790133979 us; error 33 us; skew 500 ppm
I20240502 01:53:10.134374 22594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.136631 22594 webserver.cc:460] Webserver started at http://127.20.243.2:42013/ using document root <none> and password file <none>
I20240502 01:53:10.137372 22594 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.137498 22594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.137806 22594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.140494 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/instance:
uuid: "f7462530110843e8889dc7b641b9b9a1"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.141192 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal/instance:
uuid: "f7462530110843e8889dc7b641b9b9a1"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.146466 22594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240502 01:53:10.149753 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.151047 22594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:10.151226 22594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
uuid: "f7462530110843e8889dc7b641b9b9a1"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.151397 22594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.180570 22594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:10.181484 22594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.181792 22594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.182809 22594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.184298 22594 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.184376 22594 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.184504 22594 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.184556 22594 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.216572 22594 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.2:41523
I20240502 01:53:10.216624 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.2:41523 every 8 connection(s)
I20240502 01:53:10.217902 22594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/data/info.pb
I20240502 01:53:10.227833 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 22594
I20240502 01:53:10.228057 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-1/wal/instance
I20240502 01:53:10.231101 22725 heartbeater.cc:347] Connected to a master server at 127.20.243.62:41431
I20240502 01:53:10.231422 22725 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.231699 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.3:0
--local_ip_for_outbound_sockets=127.20.243.3
--webserver_interface=127.20.243.3
--webserver_port=0
--tserver_master_addrs=127.20.243.62:41431
--builtin_ntp_servers=127.20.243.20:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 01:53:10.232240 22725 heartbeater.cc:511] Master 127.20.243.62:41431 requested a full tablet report, sending...
I20240502 01:53:10.233723 22397 ts_manager.cc:194] Registered new tserver with Master: f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523)
I20240502 01:53:10.234975 22397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.2:35539
W20240502 01:53:10.568946 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.569249 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.569355 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.577299 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.577500 22728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.3
I20240502 01:53:10.587193 22728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.20.243.3
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.243.62:41431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.3
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.588567 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.590409 22728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.602988 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.603643 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.604135 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.604671 22728 server_base.cc:1034] running on GCE node
I20240502 01:53:10.605309 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.606346 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.607594 22728 hybrid_clock.cc:648] HybridClock initialized: now 1714614790607525 us; error 46 us; skew 500 ppm
I20240502 01:53:10.607904 22728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.613948 22728 webserver.cc:460] Webserver started at http://127.20.243.3:37337/ using document root <none> and password file <none>
I20240502 01:53:10.614763 22728 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.614912 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.615223 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.619841 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/instance:
uuid: "785f3b6c508e4b80ae6f16a408b1be89"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.620661 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal/instance:
uuid: "785f3b6c508e4b80ae6f16a408b1be89"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.626353 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 01:53:10.630053 22742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.631609 22728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:10.631788 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
uuid: "785f3b6c508e4b80ae6f16a408b1be89"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-l6m7"
I20240502 01:53:10.631989 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.671176 22728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:10.672070 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.672394 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.673465 22728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.675211 22728 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.675307 22728 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.675446 22728 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.675503 22728 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.707798 22728 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.3:39403
I20240502 01:53:10.707849 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.3:39403 every 8 connection(s)
I20240502 01:53:10.709182 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/data/info.pb
I20240502 01:53:10.715934 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 22728
I20240502 01:53:10.716171 21452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-2/wal/instance
I20240502 01:53:10.722852 22856 heartbeater.cc:347] Connected to a master server at 127.20.243.62:41431
I20240502 01:53:10.723170 22856 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.724143 22856 heartbeater.cc:511] Master 127.20.243.62:41431 requested a full tablet report, sending...
I20240502 01:53:10.725637 22397 ts_manager.cc:194] Registered new tserver with Master: 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
I20240502 01:53:10.726475 22397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.3:55719
I20240502 01:53:10.732393 21452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:53:10.747866 22397 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39874:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 01:53:10.753584 22397 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:53:10.766456 22590 heartbeater.cc:503] Master 127.20.243.62:41431 was elected leader, sending a full tablet report...
I20240502 01:53:10.781179 22657 tablet_service.cc:1467] Processing CreateTablet for tablet 844d4d6937e84c2c87ce64b79326e3a3 (DEFAULT_TABLE table=TestTable [id=f0d8fa9565174a979249afe151af3daa]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:53:10.782887 22657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844d4d6937e84c2c87ce64b79326e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:10.782912 22790 tablet_service.cc:1467] Processing CreateTablet for tablet 844d4d6937e84c2c87ce64b79326e3a3 (DEFAULT_TABLE table=TestTable [id=f0d8fa9565174a979249afe151af3daa]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:53:10.784077 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844d4d6937e84c2c87ce64b79326e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:10.789520 22523 tablet_service.cc:1467] Processing CreateTablet for tablet 844d4d6937e84c2c87ce64b79326e3a3 (DEFAULT_TABLE table=TestTable [id=f0d8fa9565174a979249afe151af3daa]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 01:53:10.791035 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844d4d6937e84c2c87ce64b79326e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:10.795174 22875 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Bootstrap starting.
I20240502 01:53:10.797351 22876 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Bootstrap starting.
I20240502 01:53:10.799274 22875 tablet_bootstrap.cc:654] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:10.801061 22876 tablet_bootstrap.cc:654] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:10.801604 22875 log.cc:824] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:10.802386 22876 log.cc:824] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:10.804667 22879 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Bootstrap starting.
I20240502 01:53:10.805284 22875 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: No bootstrap required, opened a new log
I20240502 01:53:10.805450 22876 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: No bootstrap required, opened a new log
I20240502 01:53:10.805567 22875 ts_tablet_manager.cc:1367] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 01:53:10.805720 22876 ts_tablet_manager.cc:1367] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:53:10.808198 22879 tablet_bootstrap.cc:654] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:10.810482 22879 log.cc:824] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:10.812325 22876 raft_consensus.cc:348] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.812685 22876 raft_consensus.cc:374] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:10.812808 22876 raft_consensus.cc:729] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7462530110843e8889dc7b641b9b9a1, State: Initialized, Role: FOLLOWER
I20240502 01:53:10.813416 22876 consensus_queue.cc:260] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.813575 22879 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: No bootstrap required, opened a new log
I20240502 01:53:10.813926 22879 ts_tablet_manager.cc:1367] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:53:10.814469 22725 heartbeater.cc:503] Master 127.20.243.62:41431 was elected leader, sending a full tablet report...
I20240502 01:53:10.814128 22875 raft_consensus.cc:348] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.814620 22875 raft_consensus.cc:374] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:10.814745 22875 raft_consensus.cc:729] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785f3b6c508e4b80ae6f16a408b1be89, State: Initialized, Role: FOLLOWER
I20240502 01:53:10.815412 22875 consensus_queue.cc:260] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.816644 22856 heartbeater.cc:503] Master 127.20.243.62:41431 was elected leader, sending a full tablet report...
I20240502 01:53:10.818568 22875 ts_tablet_manager.cc:1398] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240502 01:53:10.819046 22876 ts_tablet_manager.cc:1398] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240502 01:53:10.823506 22879 raft_consensus.cc:348] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.823884 22879 raft_consensus.cc:374] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:10.823982 22879 raft_consensus.cc:729] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c3dd1ba6ee4febad9e9fdf89a6b3b4, State: Initialized, Role: FOLLOWER
I20240502 01:53:10.824534 22879 consensus_queue.cc:260] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.826172 22879 ts_tablet_manager.cc:1398] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240502 01:53:10.952221 22883 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:10.952536 22883 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.954398 22883 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523), 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
W20240502 01:53:10.961009 22857 tablet.cc:2369] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:10.962217 22810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785f3b6c508e4b80ae6f16a408b1be89" is_pre_election: true
I20240502 01:53:10.962224 22678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7462530110843e8889dc7b641b9b9a1" is_pre_election: true
I20240502 01:53:10.962775 22810 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c3dd1ba6ee4febad9e9fdf89a6b3b4 in term 0.
I20240502 01:53:10.962781 22678 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c3dd1ba6ee4febad9e9fdf89a6b3b4 in term 0.
I20240502 01:53:10.963413 22477 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 785f3b6c508e4b80ae6f16a408b1be89, c6c3dd1ba6ee4febad9e9fdf89a6b3b4, f7462530110843e8889dc7b641b9b9a1; no voters:
I20240502 01:53:10.963815 22883 raft_consensus.cc:2781] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:10.963987 22883 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:10.964097 22883 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:10.966120 22883 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.966758 22883 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 1 election: Requested vote from peers f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523), 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
I20240502 01:53:10.967188 22678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7462530110843e8889dc7b641b9b9a1"
I20240502 01:53:10.967275 22810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785f3b6c508e4b80ae6f16a408b1be89"
I20240502 01:53:10.967386 22678 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:10.967480 22810 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:10.969534 22678 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c3dd1ba6ee4febad9e9fdf89a6b3b4 in term 1.
I20240502 01:53:10.969540 22810 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c3dd1ba6ee4febad9e9fdf89a6b3b4 in term 1.
I20240502 01:53:10.969933 22477 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4, f7462530110843e8889dc7b641b9b9a1; no voters:
W20240502 01:53:10.970043 22726 tablet.cc:2369] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:10.970237 22883 raft_consensus.cc:2781] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:10.970769 22883 raft_consensus.cc:686] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 LEADER]: Becoming Leader. State: Replica: c6c3dd1ba6ee4febad9e9fdf89a6b3b4, State: Running, Role: LEADER
I20240502 01:53:10.971395 22883 consensus_queue.cc:237] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:10.976373 22397 catalog_manager.cc:5484] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1). New cstate: current_term: 1 leader_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } health_report { overall_health: HEALTHY } } }
W20240502 01:53:10.996361 22591 tablet.cc:2369] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:11.010775 21452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:53:11.012827 21452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c6c3dd1ba6ee4febad9e9fdf89a6b3b4 to finish bootstrapping
I20240502 01:53:11.021764 21452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f7462530110843e8889dc7b641b9b9a1 to finish bootstrapping
I20240502 01:53:11.029058 21452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 785f3b6c508e4b80ae6f16a408b1be89 to finish bootstrapping
I20240502 01:53:11.036456 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 22458
I20240502 01:53:11.037453 22585 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:11.125022 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 22458
I20240502 01:53:11.146837 21452 linked_list-test-util.h:401] Running for: 30.000s
I20240502 01:53:11.146968 21452 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 01:53:11.147042 21452 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
W20240502 01:53:11.155236 22860 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495) has failed: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111)
W20240502 01:53:11.171852 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.186259 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.200631 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.207705 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.228283 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.238360 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.264308 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.280424 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.311385 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.329568 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.369946 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.390172 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.434376 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.457646 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.506970 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.534358 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.591712 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.623003 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.688429 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.719802 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.790315 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.825446 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.903103 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:11.940640 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.025733 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.070266 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.155524 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.199859 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.245673 22860 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495) has failed: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111)
W20240502 01:53:12.292569 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.338855 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.443287 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.494729 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.599292 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.651770 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
I20240502 01:53:12.752369 22893 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:12.752585 22893 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:12.754175 22893 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403), c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495)
W20240502 01:53:12.755901 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111)
W20240502 01:53:12.757367 22610 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111)
I20240502 01:53:12.760313 22810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "f7462530110843e8889dc7b641b9b9a1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785f3b6c508e4b80ae6f16a408b1be89" is_pre_election: true
I20240502 01:53:12.760571 22810 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7462530110843e8889dc7b641b9b9a1 in term 1.
I20240502 01:53:12.761035 22610 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 785f3b6c508e4b80ae6f16a408b1be89, f7462530110843e8889dc7b641b9b9a1; no voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:12.761422 22893 raft_consensus.cc:2781] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:53:12.761555 22893 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:12.761662 22893 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:12.763085 22636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43414: Illegal state: replica f7462530110843e8889dc7b641b9b9a1 is not leader of this config: current role FOLLOWER
I20240502 01:53:12.764032 22893 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:12.764660 22893 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 election: Requested vote from peers 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403), c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495)
I20240502 01:53:12.765208 22810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844d4d6937e84c2c87ce64b79326e3a3" candidate_uuid: "f7462530110843e8889dc7b641b9b9a1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785f3b6c508e4b80ae6f16a408b1be89"
I20240502 01:53:12.765503 22810 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:12.766175 22610 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111)
I20240502 01:53:12.767647 22810 raft_consensus.cc:2454] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7462530110843e8889dc7b641b9b9a1 in term 2.
I20240502 01:53:12.768051 22610 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 785f3b6c508e4b80ae6f16a408b1be89, f7462530110843e8889dc7b641b9b9a1; no voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:12.768324 22893 raft_consensus.cc:2781] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:12.768718 22893 raft_consensus.cc:686] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [term 2 LEADER]: Becoming Leader. State: Replica: f7462530110843e8889dc7b641b9b9a1, State: Running, Role: LEADER
I20240502 01:53:12.769290 22893 consensus_queue.cc:237] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:12.773589 22397 catalog_manager.cc:5484] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 reported cstate change: term changed from 1 to 2, leader changed from c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1) to f7462530110843e8889dc7b641b9b9a1 (127.20.243.2). New cstate: current_term: 2 leader_uuid: "f7462530110843e8889dc7b641b9b9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:12.822417 22770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47198: Illegal state: replica 785f3b6c508e4b80ae6f16a408b1be89 is not leader of this config: current role FOLLOWER
W20240502 01:53:12.887403 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:12.887480 22810 raft_consensus.cc:1261] T 844d4d6937e84c2c87ce64b79326e3a3 P 785f3b6c508e4b80ae6f16a408b1be89 [term 2 FOLLOWER]: Refusing update from remote peer f7462530110843e8889dc7b641b9b9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:53:12.888319 22898 consensus_queue.cc:1035] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:15.172948 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:53:17.664988 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:53:18.150277 22893 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 35 ops behind the committed index
W20240502 01:53:18.199433 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 01:53:20.266611 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:53:21.147764 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062225472319488 Inserted count: 41650
I20240502 01:53:21.334612 22893 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 353 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:53:22.857482 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:23.364435 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 01:53:24.674793 22893 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 680 ops behind the committed index [suppressed 6 similar messages]
W20240502 01:53:25.088428 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:27.926919 22946 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 996 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:53:27.936914 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:28.559625 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:53:30.815755 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:53:31.159315 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062266486243328 Inserted count: 90650
I20240502 01:53:31.304852 22893 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 1340 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:53:33.343210 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:33.883196 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 01:53:34.477262 22946 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 1662 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:53:36.297019 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:37.775018 22943 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 1981 ops behind the committed index [suppressed 5 similar messages]
W20240502 01:53:38.970347 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 01:53:38.971491 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:53:41.116894 22943 consensus_queue.cc:786] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 [LEADER]: Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 is lagging by at least 2321 ops behind the committed index [suppressed 5 similar messages]
I20240502 01:53:41.146950 21452 linked_list-test-util.h:417] Sample at HT timestamp: 7023062307401981952 Inserted count: 140100
I20240502 01:53:41.147197 21452 linked_list-test-util.h:421] Finished inserting list. Added 140100 in chain
I20240502 01:53:41.147303 21452 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 01:53:41.147387 21452 linked_list-test-util.h:424] 0: -5309975476268826624
I20240502 01:53:41.147449 21452 linked_list-test-util.h:424] 1: -7936363378722275327
I20240502 01:53:41.147516 21452 linked_list-test-util.h:424] 2: 7884133530022248450
I20240502 01:53:41.147578 21452 linked_list-test-util.h:424] 3: 5258027102545510403
I20240502 01:53:41.147647 21452 linked_list-test-util.h:424] 4: -2678195538688344060
I20240502 01:53:41.147704 21452 linked_list-test-util.h:424] 5: -5304583441141792763
I20240502 01:53:41.147807 21452 linked_list-test-util.h:424] 6: -7930830606106820602
I20240502 01:53:41.147881 21452 linked_list-test-util.h:424] 7: 7889807040125992967
I20240502 01:53:41.147953 21452 linked_list-test-util.h:424] 8: -46415601107861496
I20240502 01:53:41.148008 21452 linked_list-test-util.h:424] 9: -2672803503561310199
I20240502 01:53:41.148077 21452 linked_list-test-util.h:424] 10: -5298909931038048246
I20240502 01:53:41.148130 21452 linked_list-test-util.h:424] 11: -7925157096003076085
I20240502 01:53:41.148197 21452 linked_list-test-util.h:424] 12: 2585364336472621068
I20240502 01:53:41.148252 21452 linked_list-test-util.h:424] 13: -41023565980827635
I20240502 01:53:41.148329 21452 linked_list-test-util.h:424] 14: -2667129993457565682
I20240502 01:53:41.148391 21452 linked_list-test-util.h:424] 15: -5293658633399304177
I20240502 01:53:41.148459 21452 linked_list-test-util.h:424] 16: 5216862799076392976
I20240502 01:53:41.148514 21452 linked_list-test-util.h:424] 17: 2590756371599654929
I20240502 01:53:41.148581 21452 linked_list-test-util.h:424] 18: -35350055877083118
I20240502 01:53:41.148640 21452 linked_list-test-util.h:424] 19: -2661878695818821613
I20240502 01:53:41.148710 21452 linked_list-test-util.h:424] 20: 7848783474145165332
I20240502 01:53:41.148815 21452 linked_list-test-util.h:424] 21: 5222536309180137493
I20240502 01:53:41.148926 21452 linked_list-test-util.h:424] 22: 2596429881703399446
I20240502 01:53:41.148994 21452 linked_list-test-util.h:424] 23: -30098758238339049
I20240502 01:53:41.149061 21452 linked_list-test-util.h:424] 24: -7966180661983903720
I20240502 01:53:41.149117 21452 linked_list-test-util.h:424] 25: 7854316246760620057
I20240502 01:53:41.149189 21452 linked_list-test-util.h:424] 26: 5227928344307171354
I20240502 01:53:41.149247 21452 linked_list-test-util.h:424] 27: -2708294296926683109
I20240502 01:53:41.149315 21452 linked_list-test-util.h:424] 28: -5334400724403421156
I20240502 01:53:41.149376 21452 linked_list-test-util.h:424] 29: -7960647889368448995
I20240502 01:53:41.149446 21452 linked_list-test-util.h:424] 30: 7859708281887653918
I20240502 01:53:41.149500 21452 linked_list-test-util.h:424] 31: -76514359346200545
I20240502 01:53:41.149574 21452 linked_list-test-util.h:424] 32: -2702620786822938592
I20240502 01:53:41.149628 21452 linked_list-test-util.h:424] 33: -5328867951787966431
I20240502 01:53:41.149701 21452 linked_list-test-util.h:424] 34: -7955255854241415134
I20240502 01:53:41.149755 21452 linked_list-test-util.h:424] 35: 2555265578234282019
I20240502 01:53:41.149832 21452 linked_list-test-util.h:424] 36: -70840849242456028
I20240502 01:53:41.149886 21452 linked_list-test-util.h:424] 37: -2697088014207483867
I20240502 01:53:41.149955 21452 linked_list-test-util.h:424] 38: -5323475916660932570
I20240502 01:53:41.150008 21452 linked_list-test-util.h:424] 39: 5187045515814764583
I20240502 01:53:41.150076 21452 linked_list-test-util.h:424] 40: 2560939088338026536
I20240502 01:53:41.150131 21452 linked_list-test-util.h:424] 41: -65448814115422167
I20240502 01:53:41.150202 21452 linked_list-test-util.h:424] 42: -2691695979080450006
I20240502 01:53:41.150255 21452 linked_list-test-util.h:424] 43: 7818825453395247147
I20240502 01:53:41.150343 21452 linked_list-test-util.h:424] 44: 5192719025918509100
I20240502 01:53:41.150401 21452 linked_list-test-util.h:424] 45: 2566331123465060397
I20240502 01:53:41.150470 21452 linked_list-test-util.h:424] 46: -59916041499967442
I20240502 01:53:41.150525 21452 linked_list-test-util.h:424] 47: -7996138682733821905
I20240502 01:53:41.150575 21452 linked_list-test-util.h:424] 48: 7824498963498991664
I20240502 01:53:41.150609 21452 linked_list-test-util.h:424] 49: 5198111061045542961
I20240502 01:53:41.245215 21452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
/tmp/dist-test-taskCWKnjP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.243.1:34495
--local_ip_for_outbound_sockets=127.20.243.1
--tserver_master_addrs=127.20.243.62:41431
--webserver_port=39805
--webserver_interface=127.20.243.1
--builtin_ntp_servers=127.20.243.20:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 01:53:41.584672 23083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:41.584985 23083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:41.585135 23083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:41.593475 23083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:41.593724 23083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.243.1
I20240502 01:53:41.603518 23083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.243.20:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.243.1:34495
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.20.243.1
--webserver_port=39805
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.20.243.62:41431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.243.1
--log_dir=/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:41.604934 23083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:41.606669 23083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:41.619020 23091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:41.619534 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:41.620265 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:41.620859 23083 server_base.cc:1034] running on GCE node
I20240502 01:53:41.621557 23083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:41.622565 23083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:41.623787 23083 hybrid_clock.cc:648] HybridClock initialized: now 1714614821623716 us; error 52 us; skew 500 ppm
I20240502 01:53:41.624091 23083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:41.637308 23083 webserver.cc:460] Webserver started at http://127.20.243.1:39805/ using document root <none> and password file <none>
I20240502 01:53:41.638379 23083 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:41.638556 23083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20240502 01:53:41.645030 22610 consensus_peers.cc:485] T 844d4d6937e84c2c87ce64b79326e3a3 P f7462530110843e8889dc7b641b9b9a1 -> Peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495): Couldn't send request to peer c6c3dd1ba6ee4febad9e9fdf89a6b3b4. Status: Network error: Client connection negotiation failed: client connection to 127.20.243.1:34495: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:53:41.645550 23083 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:53:41.648432 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:41.649744 23083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:41.649904 23083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-l6m7"
I20240502 01:53:41.650815 23083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:41.674070 23083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:41.674997 23083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:41.675320 23083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:41.676362 23083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:41.678364 23106 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:53:41.683984 23083 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:53:41.684115 23083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240502 01:53:41.684247 23083 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:53:41.687129 23083 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:53:41.687211 23083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:41.687541 23106 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Bootstrap starting.
I20240502 01:53:41.707100 23106 log.cc:824] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:41.711621 23106 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:41.712066 23106 log.cc:1225] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/wal/wals/844d4d6937e84c2c87ce64b79326e3a3.recovery-1714614821711926
I20240502 01:53:41.712317 23106 tablet_bootstrap.cc:492] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Bootstrap complete.
I20240502 01:53:41.712997 23106 ts_tablet_manager.cc:1367] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.012s
I20240502 01:53:41.724220 23106 raft_consensus.cc:348] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:41.726347 23106 raft_consensus.cc:729] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c3dd1ba6ee4febad9e9fdf89a6b3b4, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.727188 23106 consensus_queue.cc:260] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:41.728888 23106 ts_tablet_manager.cc:1398] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20240502 01:53:41.740893 23083 rpc_server.cc:307] RPC server started. Bound to: 127.20.243.1:34495
I20240502 01:53:41.740932 23217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.243.1:34495 every 8 connection(s)
I20240502 01:53:41.742597 23083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCWKnjP/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714614750802054-21452-0/linked-list-cluster/ts-0/data/info.pb
W20240502 01:53:41.743851 23219 tablet.cc:2369] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:41.750916 21452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu as pid 23083
I20240502 01:53:41.754489 23218 heartbeater.cc:347] Connected to a master server at 127.20.243.62:41431
I20240502 01:53:41.754899 23218 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:41.756110 23218 heartbeater.cc:511] Master 127.20.243.62:41431 requested a full tablet report, sending...
I20240502 01:53:41.759328 22398 ts_manager.cc:194] Re-registered known tserver with Master: c6c3dd1ba6ee4febad9e9fdf89a6b3b4 (127.20.243.1:34495)
I20240502 01:53:41.760679 22398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.243.1:44015
I20240502 01:53:41.777390 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 1.1 2.2831 2.2831
I20240502 01:53:41.880882 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 1.1 2.2831 2.2831
I20240502 01:53:42.084283 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 1.1 2.2831 2.2831
I20240502 01:53:42.207679 23171 raft_consensus.cc:3037] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:42.211746 23171 pending_rounds.cc:76] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Aborting all ops after (but not including) 0
I20240502 01:53:42.211877 23171 pending_rounds.cc:97] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 01:53:42.212038 23171 raft_consensus.cc:2866] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:53:42.212713 23218 heartbeater.cc:503] Master 127.20.243.62:41431 was elected leader, sending a full tablet report...
I20240502 01:53:42.390182 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 2.505 2.2831 2.2831
I20240502 01:53:42.817392 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 2.1457 2.2831 2.2831
I20240502 01:53:43.084731 23228 mvcc.cc:205] Tried to move back new op lower bound from 7023062211791343616 to 7023062191186743296. Current Snapshot: MvccSnapshot[applied={T|T < 7023062208664064000 or (T in {7023062208670121984,7023062208777945088,7023062208828391424,7023062208873496576,7023062208951705600,7023062209013243904,7023062209110814720,7023062209200148480,7023062209244786688,7023062209155870720,7023062209286012928,7023062209325461504,7023062209364664320,7023062209406656512,7023062209447014400,7023062209482289152,7023062209515331584,7023062209547247616,7023062209614336000,7023062209645699072,7023062209677660160,7023062209583149056,7023062209713745920,7023062209747144704,7023062209781248000,7023062209819287552,7023062209855696896,7023062209889226752,7023062209963663360,7023062210002534400,7023062210040360960,7023062209926836224,7023062210113576960,7023062210076577792,7023062210157727744,7023062210197172224,7023062210233745408,7023062210270089216,7023062210389970944,7023062210312486912,7023062210426232832,7023062210461945856,7023062210352525312,7023062210529517568,7023062210497359872,7023062210563641344,7023062210630373376,7023062210597048320,7023062210705297408,7023062210741317632,7023062210663723008,7023062210815631360,7023062210776776704,7023062210851287040,7023062210958036992,7023062210997362688,7023062211035992064,7023062211072319488,7023062210920763392,7023062210888056832,7023062211103498240,7023062211136331776,7023062211206791168,7023062211245031424,7023062211277770752,7023062211313045504,7023062211348172800,7023062211381338112,7023062211451404288,7023062211413757952,7023062211488903168,7023062211521908736,7023062211173232640,7023062211594428416,7023062211557740544,7023062211628179456})}]
I20240502 01:53:43.334194 21452 cluster_itest_util.cc:256] Not converged past 2802 yet: 2.2455 2.2831 2.2831
I20240502 01:53:43.938027 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 22594
I20240502 01:53:43.939145 22720 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:44.277300 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 22594
W20240502 01:53:44.339448 23102 connection.cc:535] server connection from 127.20.243.2:38677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:44.339936 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 22728
I20240502 01:53:44.341248 22851 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:44.665851 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 22728
I20240502 01:53:44.722163 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614801140703 usec, L: 1 Expected Rows: 41650
I20240502 01:53:44.724005 21452 meta_cache.cc:1510] marking tablet server f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) as failed
W20240502 01:53:44.724211 21452 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:44.727512 21452 meta_cache.cc:1510] marking tablet server 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403) as failed
I20240502 01:53:44.777951 21452 linked_list-test-util.h:528] Calling callback on tserver c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:45.789615 23243 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7462530110843e8889dc7b641b9b9a1)
I20240502 01:53:45.789933 23243 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:45.796130 23243 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523), 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
W20240502 01:53:45.796983 23101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.3:39403: connect: Connection refused (error 111)
W20240502 01:53:45.796983 23103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
W20240502 01:53:45.798521 23101 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403): Network error: Client connection negotiation failed: client connection to 127.20.243.3:39403: connect: Connection refused (error 111)
W20240502 01:53:45.799221 23103 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523): Network error: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:45.799332 23103 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4; no voters: 785f3b6c508e4b80ae6f16a408b1be89, f7462530110843e8889dc7b641b9b9a1
I20240502 01:53:45.799705 23243 raft_consensus.cc:2726] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:53:46.528304 21452 linked_list-test-util.h:767] Done collecting results (41650 rows in 1750.52ms)
I20240502 01:53:46.612593 21452 linked_list-test-util.h:556] Snapshot: P: 1714614801140703 usec, L: 1 verified. Result: OK
I20240502 01:53:46.616005 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614811153868 usec, L: 1 Expected Rows: 90650
I20240502 01:53:46.617604 21452 meta_cache.cc:1510] marking tablet server f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) as failed
W20240502 01:53:46.617722 21452 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:46.620440 21452 meta_cache.cc:1510] marking tablet server 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403) as failed
I20240502 01:53:46.627069 21452 linked_list-test-util.h:528] Calling callback on tserver c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:47.635536 23248 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7462530110843e8889dc7b641b9b9a1)
I20240502 01:53:47.635713 23248 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:47.637292 23248 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523), 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
W20240502 01:53:47.638638 23101 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403): Network error: Client connection negotiation failed: client connection to 127.20.243.3:39403: connect: Connection refused (error 111)
W20240502 01:53:47.639340 23103 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523): Network error: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:47.639433 23103 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4; no voters: 785f3b6c508e4b80ae6f16a408b1be89, f7462530110843e8889dc7b641b9b9a1
I20240502 01:53:47.639864 23248 raft_consensus.cc:2726] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:53:48.672680 21452 linked_list-test-util.h:767] Done collecting results (90650 rows in 2045.72ms)
I20240502 01:53:48.781677 21452 linked_list-test-util.h:556] Snapshot: P: 1714614811153868 usec, L: 1 verified. Result: OK
I20240502 01:53:48.784947 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 1714614821143062 usec, L: 1 Expected Rows: 140100
I20240502 01:53:48.786602 21452 meta_cache.cc:1510] marking tablet server f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) as failed
W20240502 01:53:48.786738 21452 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:48.789145 21452 meta_cache.cc:1510] marking tablet server 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403) as failed
I20240502 01:53:48.797036 21452 linked_list-test-util.h:528] Calling callback on tserver c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:49.754933 23253 raft_consensus.cc:482] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7462530110843e8889dc7b641b9b9a1)
I20240502 01:53:49.755095 23253 raft_consensus.cc:504] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7462530110843e8889dc7b641b9b9a1" member_type: VOTER last_known_addr { host: "127.20.243.2" port: 41523 } } peers { permanent_uuid: "785f3b6c508e4b80ae6f16a408b1be89" member_type: VOTER last_known_addr { host: "127.20.243.3" port: 39403 } } peers { permanent_uuid: "c6c3dd1ba6ee4febad9e9fdf89a6b3b4" member_type: VOTER last_known_addr { host: "127.20.243.1" port: 34495 } }
I20240502 01:53:49.755720 23253 leader_election.cc:288] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523), 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403)
W20240502 01:53:49.757577 23103 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523): Network error: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
W20240502 01:53:49.757767 23101 leader_election.cc:334] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403): Network error: Client connection negotiation failed: client connection to 127.20.243.3:39403: connect: Connection refused (error 111)
I20240502 01:53:49.757899 23101 leader_election.cc:302] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c3dd1ba6ee4febad9e9fdf89a6b3b4; no voters: 785f3b6c508e4b80ae6f16a408b1be89, f7462530110843e8889dc7b641b9b9a1
I20240502 01:53:49.758128 23253 raft_consensus.cc:2726] T 844d4d6937e84c2c87ce64b79326e3a3 P c6c3dd1ba6ee4febad9e9fdf89a6b3b4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:53:50.777627 21452 linked_list-test-util.h:767] Done collecting results (140100 rows in 1980.72ms)
I20240502 01:53:50.948093 21452 linked_list-test-util.h:556] Snapshot: P: 1714614821143062 usec, L: 1 verified. Result: OK
I20240502 01:53:50.951531 21452 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 140100
I20240502 01:53:50.953161 21452 meta_cache.cc:1510] marking tablet server f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) as failed
W20240502 01:53:50.953275 21452 meta_cache.cc:302] tablet 844d4d6937e84c2c87ce64b79326e3a3: replica f7462530110843e8889dc7b641b9b9a1 (127.20.243.2:41523) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.20.243.2:41523: connect: Connection refused (error 111)
I20240502 01:53:50.955905 21452 meta_cache.cc:1510] marking tablet server 785f3b6c508e4b80ae6f16a408b1be89 (127.20.243.3:39403) as failed
I20240502 01:53:50.958555 21452 linked_list-test-util.h:528] Calling callback on tserver c6c3dd1ba6ee4febad9e9fdf89a6b3b4
I20240502 01:53:52.901465 21452 linked_list-test-util.h:767] Done collecting results (140100 rows in 1943.01ms)
I20240502 01:53:53.074383 21452 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240502 01:53:53.075326 21452 linked_list-test-util.h:692] Successfully verified 140100 rows
I20240502 01:53:53.076493 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 23083
I20240502 01:53:53.088793 23213 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:53.330976 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 23083
I20240502 01:53:53.379014 21452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu pid 22366
I20240502 01:53:53.380232 22429 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:53.493878 21452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCWKnjP/build/asan/bin/kudu with pid 22366
2024-05-02T01:53:53Z chronyd exiting
[ OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (44861 ms)
[----------] 2 tests from LinkedListTest (82672 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (82673 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] LinkedListTest.TestLoadAndVerify
1 FAILED TEST
I20240502 01:53:53.542806 21452 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~2 seconds ago