[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:39.328747 15550 test_util.cc:268] Using random seed: 1050779283
I20240502 21:25:39.343751 15550 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 21:25:39.343850 15550 linked_list-test.cc:101] --------------
I20240502 21:25:39.343907 15550 linked_list-test.cc:102] 50 chains
I20240502 21:25:39.343979 15550 linked_list-test.cc:103] 3 tablets
I20240502 21:25:39.344051 15550 linked_list-test.cc:104] Mutations on
I20240502 21:25:39.344107 15550 linked_list-test.cc:105] --------------
I20240502 21:25:39.344444 15550 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:39.344564 15550 ts_itest-base.cc:113] --------------
I20240502 21:25:39.344694 15550 ts_itest-base.cc:114] 3 tablet servers
I20240502 21:25:39.344794 15550 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:39.344924 15550 ts_itest-base.cc:116] --------------
2024-05-02T21:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:39Z Disabled control of system clock
I20240502 21:25:39.373975 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.47.190:46459
--webserver_interface=127.15.47.190
--webserver_port=0
--builtin_ntp_servers=127.15.47.148:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.47.190:46459
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 21:25:39.771057 15562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.771395 15562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.771546 15562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.779145 15562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:39.779305 15562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.779399 15562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:39.779482 15562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:39.788733 15562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.47.190:46459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.47.190:46459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.15.47.190
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.790184 15562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.792002 15562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.803576 15568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.804052 15567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.804848 15570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.805444 15562 server_base.cc:1034] running on GCE node
I20240502 21:25:39.806269 15562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.807754 15562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.809002 15562 hybrid_clock.cc:648] HybridClock initialized: now 1714685139808939 us; error 37 us; skew 500 ppm
I20240502 21:25:39.809337 15562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.815418 15562 webserver.cc:460] Webserver started at http://127.15.47.190:39895/ using document root <none> and password file <none>
I20240502 21:25:39.816229 15562 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.816412 15562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.816756 15562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.819614 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/instance:
uuid: "ee247959788c41c6ab76c90851302110"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-ntzt"
I20240502 21:25:39.820470 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal/instance:
uuid: "ee247959788c41c6ab76c90851302110"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-ntzt"
I20240502 21:25:39.825982 15562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240502 21:25:39.829530 15576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.830982 15562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:39.831183 15562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
uuid: "ee247959788c41c6ab76c90851302110"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-ntzt"
I20240502 21:25:39.831403 15562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.877583 15562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.878538 15562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.878845 15562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.897832 15562 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.190:46459
I20240502 21:25:39.897852 15627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.190:46459 every 8 connection(s)
I20240502 21:25:39.899322 15562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
I20240502 21:25:39.903244 15628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.904083 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 15562
I20240502 21:25:39.904415 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal/instance
I20240502 21:25:39.915655 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110: Bootstrap starting.
I20240502 21:25:39.918771 15628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.919831 15628 log.cc:824] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.922945 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110: No bootstrap required, opened a new log
I20240502 21:25:39.929409 15628 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } }
I20240502 21:25:39.929780 15628 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.929898 15628 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee247959788c41c6ab76c90851302110, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.930552 15628 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } }
I20240502 21:25:39.930766 15628 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:39.930922 15628 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:39.931099 15628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.933883 15628 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } }
I20240502 21:25:39.934478 15628 leader_election.cc:302] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee247959788c41c6ab76c90851302110; no voters:
I20240502 21:25:39.935195 15628 leader_election.cc:288] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:39.935370 15633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.936085 15633 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [term 1 LEADER]: Becoming Leader. State: Replica: ee247959788c41c6ab76c90851302110, State: Running, Role: LEADER
I20240502 21:25:39.936668 15633 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } }
I20240502 21:25:39.937490 15628 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:39.940155 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee247959788c41c6ab76c90851302110. Latest consensus state: current_term: 1 leader_uuid: "ee247959788c41c6ab76c90851302110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } } }
I20240502 21:25:39.940328 15634 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee247959788c41c6ab76c90851302110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee247959788c41c6ab76c90851302110" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 46459 } } }
I20240502 21:25:39.940644 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:39.940649 15634 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:39.942463 15642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:39.947504 15642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:39.956884 15642 catalog_manager.cc:1347] Generated new cluster ID: 4cdb884b55f44fbcb434de6789e75527
I20240502 21:25:39.957032 15642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:39.976712 15642 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:39.978580 15642 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:40.004884 15642 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee247959788c41c6ab76c90851302110: Generated new TSK 0
I20240502 21:25:40.006108 15642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:40.017244 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.129:0
--local_ip_for_outbound_sockets=127.15.47.129
--webserver_interface=127.15.47.129
--webserver_port=0
--tserver_master_addrs=127.15.47.190:46459
--builtin_ntp_servers=127.15.47.148:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 21:25:40.374048 15652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:40.374399 15652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:40.374527 15652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:40.382551 15652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:40.382743 15652 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 21:25:40.382792 15652 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 21:25:40.382858 15652 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 21:25:40.382944 15652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.129
I20240502 21:25:40.392957 15652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.15.47.129
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.15.47.190:46459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.129
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:40.394498 15652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:40.396679 15652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:40.412106 15657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.412635 15658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.413275 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:40.413944 15652 server_base.cc:1034] running on GCE node
I20240502 21:25:40.414816 15652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:40.416106 15652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:40.417393 15652 hybrid_clock.cc:648] HybridClock initialized: now 1714685140417344 us; error 78 us; skew 500 ppm
I20240502 21:25:40.417826 15652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:40.421182 15652 webserver.cc:460] Webserver started at http://127.15.47.129:37047/ using document root <none> and password file <none>
I20240502 21:25:40.422335 15652 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:40.422519 15652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:40.422911 15652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:40.426328 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/instance:
uuid: "7575b61d310d4850bbf16b4827e92922"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.427318 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal/instance:
uuid: "7575b61d310d4850bbf16b4827e92922"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.434042 15652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 21:25:40.438195 15666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.439867 15652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:25:40.440057 15652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
uuid: "7575b61d310d4850bbf16b4827e92922"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.440315 15652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:40.466313 15652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:40.467533 15652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:40.467872 15652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:40.469005 15652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:40.470623 15652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:40.470729 15652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.470850 15652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:40.470925 15652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.504693 15652 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.129:38827
I20240502 21:25:40.504738 15779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.129:38827 every 8 connection(s)
I20240502 21:25:40.506096 15652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
I20240502 21:25:40.508139 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 15652
I20240502 21:25:40.508378 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal/instance
I20240502 21:25:40.512526 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.130:0
--local_ip_for_outbound_sockets=127.15.47.130
--webserver_interface=127.15.47.130
--webserver_port=0
--tserver_master_addrs=127.15.47.190:46459
--builtin_ntp_servers=127.15.47.148:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 21:25:40.521238 15780 heartbeater.cc:347] Connected to a master server at 127.15.47.190:46459
I20240502 21:25:40.521566 15780 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:40.522455 15780 heartbeater.cc:511] Master 127.15.47.190:46459 requested a full tablet report, sending...
I20240502 21:25:40.525012 15593 ts_manager.cc:194] Registered new tserver with Master: 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827)
I20240502 21:25:40.527485 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.129:57631
W20240502 21:25:40.879845 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:40.880170 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:40.880381 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:40.888429 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:40.888638 15785 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 21:25:40.888705 15785 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 21:25:40.888806 15785 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 21:25:40.888955 15785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.130
I20240502 21:25:40.898602 15785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.15.47.130
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.15.47.190:46459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.130
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:40.900007 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:40.901768 15785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:40.917548 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.918115 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.918663 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:40.919298 15785 server_base.cc:1034] running on GCE node
I20240502 21:25:40.919957 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:40.921028 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:40.922257 15785 hybrid_clock.cc:648] HybridClock initialized: now 1714685140922187 us; error 65 us; skew 500 ppm
I20240502 21:25:40.922585 15785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:40.925098 15785 webserver.cc:460] Webserver started at http://127.15.47.130:44187/ using document root <none> and password file <none>
I20240502 21:25:40.925884 15785 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:40.926004 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:40.926323 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:40.929203 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/instance:
uuid: "1df45829447d4299992c186d49363fb4"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.929992 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal/instance:
uuid: "1df45829447d4299992c186d49363fb4"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.935796 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:25:40.940107 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.941869 15785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:40.942071 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
uuid: "1df45829447d4299992c186d49363fb4"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-ntzt"
I20240502 21:25:40.942282 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:40.959056 15785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:40.959990 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:40.960327 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:40.961429 15785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:40.963094 15785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:40.963182 15785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.963296 15785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:40.963371 15785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.001192 15785 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.130:35433
I20240502 21:25:41.001219 15912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.130:35433 every 8 connection(s)
I20240502 21:25:41.002633 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
I20240502 21:25:41.004506 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 15785
I20240502 21:25:41.004698 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal/instance
I20240502 21:25:41.008793 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.131:0
--local_ip_for_outbound_sockets=127.15.47.131
--webserver_interface=127.15.47.131
--webserver_port=0
--tserver_master_addrs=127.15.47.190:46459
--builtin_ntp_servers=127.15.47.148:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 21:25:41.019907 15913 heartbeater.cc:347] Connected to a master server at 127.15.47.190:46459
I20240502 21:25:41.020390 15913 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.021306 15913 heartbeater.cc:511] Master 127.15.47.190:46459 requested a full tablet report, sending...
I20240502 21:25:41.022888 15593 ts_manager.cc:194] Registered new tserver with Master: 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433)
I20240502 21:25:41.023682 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.130:40943
W20240502 21:25:41.380533 15916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:41.380897 15916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:41.381078 15916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:41.389479 15916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:41.389698 15916 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 21:25:41.389765 15916 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 21:25:41.389878 15916 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 21:25:41.390007 15916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.131
I20240502 21:25:41.399530 15916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.15.47.131
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.15.47.190:46459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.131
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:41.400997 15916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:41.402781 15916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:41.415591 15924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.416304 15921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.416805 15922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:41.417546 15916 server_base.cc:1034] running on GCE node
I20240502 21:25:41.418184 15916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:41.419193 15916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:41.420394 15916 hybrid_clock.cc:648] HybridClock initialized: now 1714685141420345 us; error 43 us; skew 500 ppm
I20240502 21:25:41.420724 15916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:41.423022 15916 webserver.cc:460] Webserver started at http://127.15.47.131:37807/ using document root <none> and password file <none>
I20240502 21:25:41.423825 15916 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:41.423950 15916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:41.424300 15916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:41.435348 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/instance:
uuid: "5e53385ddc904372a6685f865d7c9e28"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-ntzt"
I20240502 21:25:41.436249 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal/instance:
uuid: "5e53385ddc904372a6685f865d7c9e28"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-ntzt"
I20240502 21:25:41.442234 15916 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 21:25:41.445953 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.447511 15916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:25:41.447685 15916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
uuid: "5e53385ddc904372a6685f865d7c9e28"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-ntzt"
I20240502 21:25:41.447933 15916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:41.463128 15916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:41.464025 15916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:41.464341 15916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:41.465308 15916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:41.467242 15916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:41.467363 15916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.467478 15916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:41.467563 15916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.501034 15916 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.131:40499
I20240502 21:25:41.501077 16044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.131:40499 every 8 connection(s)
I20240502 21:25:41.502346 15916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
I20240502 21:25:41.504248 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 15916
I20240502 21:25:41.504503 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal/instance
I20240502 21:25:41.516166 16045 heartbeater.cc:347] Connected to a master server at 127.15.47.190:46459
I20240502 21:25:41.516499 16045 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.517300 16045 heartbeater.cc:511] Master 127.15.47.190:46459 requested a full tablet report, sending...
I20240502 21:25:41.518692 15593 ts_manager.cc:194] Registered new tserver with Master: 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.519692 15593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.131:58269
I20240502 21:25:41.521373 15550 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:25:41.530905 15780 heartbeater.cc:503] Master 127.15.47.190:46459 was elected leader, sending a full tablet report...
I20240502 21:25:41.543669 15593 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53158:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 21:25:41.550487 15593 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:25:41.582007 15977 tablet_service.cc:1467] Processing CreateTablet for tablet 23c3098e6fc94593a4adbd5b89076b2c (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 21:25:41.582798 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d451ab3b430dab914b22d4dbb83b (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 21:25:41.581859 15714 tablet_service.cc:1467] Processing CreateTablet for tablet ce42e0503d2c4b08a3a825f2d1e6fd01 (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 21:25:41.584028 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d451ab3b430dab914b22d4dbb83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.584074 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c3098e6fc94593a4adbd5b89076b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.584221 15979 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d451ab3b430dab914b22d4dbb83b (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 21:25:41.584647 15714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce42e0503d2c4b08a3a825f2d1e6fd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.584821 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d451ab3b430dab914b22d4dbb83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.585801 15978 tablet_service.cc:1467] Processing CreateTablet for tablet ce42e0503d2c4b08a3a825f2d1e6fd01 (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 21:25:41.581818 15713 tablet_service.cc:1467] Processing CreateTablet for tablet 23c3098e6fc94593a4adbd5b89076b2c (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 21:25:41.586946 15713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c3098e6fc94593a4adbd5b89076b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.587766 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce42e0503d2c4b08a3a825f2d1e6fd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.590824 15845 tablet_service.cc:1467] Processing CreateTablet for tablet 23c3098e6fc94593a4adbd5b89076b2c (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 21:25:41.593005 15845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c3098e6fc94593a4adbd5b89076b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.590991 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d451ab3b430dab914b22d4dbb83b (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 21:25:41.594843 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d451ab3b430dab914b22d4dbb83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.591147 15846 tablet_service.cc:1467] Processing CreateTablet for tablet ce42e0503d2c4b08a3a825f2d1e6fd01 (DEFAULT_TABLE table=TestTable [id=a1fae71f33ea48bc86c50e40908fd1b8]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 21:25:41.595798 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce42e0503d2c4b08a3a825f2d1e6fd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.609269 16064 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Bootstrap starting.
I20240502 21:25:41.613127 16064 tablet_bootstrap.cc:654] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.614549 16064 log.cc:824] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:41.615475 16066 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Bootstrap starting.
I20240502 21:25:41.616356 16065 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: Bootstrap starting.
I20240502 21:25:41.619676 16065 tablet_bootstrap.cc:654] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.621111 16065 log.cc:824] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:41.622206 16066 tablet_bootstrap.cc:654] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.623672 16066 log.cc:824] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:41.624068 16064 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: No bootstrap required, opened a new log
I20240502 21:25:41.624466 16064 ts_tablet_manager.cc:1367] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.008s
I20240502 21:25:41.625792 16065 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: No bootstrap required, opened a new log
I20240502 21:25:41.626061 16065 ts_tablet_manager.cc:1367] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240502 21:25:41.633038 16065 raft_consensus.cc:348] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.633424 16065 raft_consensus.cc:374] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.633523 16065 raft_consensus.cc:729] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df45829447d4299992c186d49363fb4, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.634194 16065 consensus_queue.cc:260] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.634414 16066 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: No bootstrap required, opened a new log
I20240502 21:25:41.634738 16066 ts_tablet_manager.cc:1367] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.008s
I20240502 21:25:41.635052 16064 raft_consensus.cc:348] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.635510 16064 raft_consensus.cc:374] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.635622 16064 raft_consensus.cc:729] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e53385ddc904372a6685f865d7c9e28, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.636353 16064 consensus_queue.cc:260] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.637262 15913 heartbeater.cc:503] Master 127.15.47.190:46459 was elected leader, sending a full tablet report...
I20240502 21:25:41.638149 16065 ts_tablet_manager.cc:1398] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4: Time spent starting tablet: real 0.012s user 0.007s sys 0.007s
I20240502 21:25:41.638690 16065 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: Bootstrap starting.
I20240502 21:25:41.640947 16065 tablet_bootstrap.cc:654] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.644682 16064 ts_tablet_manager.cc:1398] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20240502 21:25:41.645097 16045 heartbeater.cc:503] Master 127.15.47.190:46459 was elected leader, sending a full tablet report...
I20240502 21:25:41.645902 16066 raft_consensus.cc:348] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.646447 16066 raft_consensus.cc:374] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.646555 16066 raft_consensus.cc:729] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7575b61d310d4850bbf16b4827e92922, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.648597 16064 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28: Bootstrap starting.
I20240502 21:25:41.648483 16066 consensus_queue.cc:260] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.651309 16064 tablet_bootstrap.cc:654] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.653425 16066 ts_tablet_manager.cc:1398] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Time spent starting tablet: real 0.019s user 0.014s sys 0.006s
I20240502 21:25:41.653856 16066 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922: Bootstrap starting.
I20240502 21:25:41.654366 16065 tablet_bootstrap.cc:492] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: No bootstrap required, opened a new log
I20240502 21:25:41.654577 16065 ts_tablet_manager.cc:1367] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240502 21:25:41.655606 16066 tablet_bootstrap.cc:654] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.655915 16065 raft_consensus.cc:348] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.656208 16065 raft_consensus.cc:374] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.656342 16065 raft_consensus.cc:729] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df45829447d4299992c186d49363fb4, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.656740 16065 consensus_queue.cc:260] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.658454 16065 ts_tablet_manager.cc:1398] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:25:41.658788 16065 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4: Bootstrap starting.
I20240502 21:25:41.661157 16065 tablet_bootstrap.cc:654] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.663828 16064 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28: No bootstrap required, opened a new log
I20240502 21:25:41.663944 16066 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922: No bootstrap required, opened a new log
I20240502 21:25:41.664012 16064 ts_tablet_manager.cc:1367] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.001s
I20240502 21:25:41.664095 16066 ts_tablet_manager.cc:1367] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 21:25:41.665028 16065 tablet_bootstrap.cc:492] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4: No bootstrap required, opened a new log
I20240502 21:25:41.665196 16065 ts_tablet_manager.cc:1367] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:25:41.665078 16066 raft_consensus.cc:348] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.665078 16064 raft_consensus.cc:348] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.665297 16066 raft_consensus.cc:374] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.665297 16064 raft_consensus.cc:374] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.665397 16064 raft_consensus.cc:729] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e53385ddc904372a6685f865d7c9e28, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.665397 16066 raft_consensus.cc:729] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7575b61d310d4850bbf16b4827e92922, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.665649 16066 consensus_queue.cc:260] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.665638 16064 consensus_queue.cc:260] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.665899 16065 raft_consensus.cc:348] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.666177 16065 raft_consensus.cc:374] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.666232 16066 ts_tablet_manager.cc:1398] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:41.666236 16064 ts_tablet_manager.cc:1398] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:41.666256 16065 raft_consensus.cc:729] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df45829447d4299992c186d49363fb4, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.666500 16064 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28: Bootstrap starting.
I20240502 21:25:41.666549 16065 consensus_queue.cc:260] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.668219 16064 tablet_bootstrap.cc:654] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.669068 16066 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922: Bootstrap starting.
I20240502 21:25:41.671106 16065 ts_tablet_manager.cc:1398] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:25:41.671267 16066 tablet_bootstrap.cc:654] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.673205 16064 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28: No bootstrap required, opened a new log
I20240502 21:25:41.673339 16064 ts_tablet_manager.cc:1367] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:25:41.673868 16064 raft_consensus.cc:348] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.674067 16064 raft_consensus.cc:374] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.674122 16064 raft_consensus.cc:729] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e53385ddc904372a6685f865d7c9e28, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.674317 16064 consensus_queue.cc:260] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.674713 16064 ts_tablet_manager.cc:1398] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:25:41.675685 16066 tablet_bootstrap.cc:492] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922: No bootstrap required, opened a new log
I20240502 21:25:41.675845 16066 ts_tablet_manager.cc:1367] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:25:41.676435 16066 raft_consensus.cc:348] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.676656 16066 raft_consensus.cc:374] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.676731 16066 raft_consensus.cc:729] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7575b61d310d4850bbf16b4827e92922, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.677001 16066 consensus_queue.cc:260] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.677510 16066 ts_tablet_manager.cc:1398] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:41.678221 16073 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.678442 16073 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.680375 16073 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.688882 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4" is_pre_election: true
I20240502 21:25:41.688875 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:41.689496 15867 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 0.
I20240502 21:25:41.689496 15999 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 0.
I20240502 21:25:41.690136 15669 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e53385ddc904372a6685f865d7c9e28, 7575b61d310d4850bbf16b4827e92922; no voters:
I20240502 21:25:41.690474 16073 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.690637 16073 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.690737 16073 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.693006 16073 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.693532 16073 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 election: Requested vote from peers 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.694018 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4"
I20240502 21:25:41.694108 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:41.694205 15867 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.694309 15999 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.696445 15867 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 1.
I20240502 21:25:41.696480 15999 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 1.
I20240502 21:25:41.696822 15669 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 7575b61d310d4850bbf16b4827e92922; no voters:
I20240502 21:25:41.697142 16073 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.697624 16073 raft_consensus.cc:686] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [term 1 LEADER]: Becoming Leader. State: Replica: 7575b61d310d4850bbf16b4827e92922, State: Running, Role: LEADER
I20240502 21:25:41.698101 16073 consensus_queue.cc:237] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.702595 15592 catalog_manager.cc:5484] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7575b61d310d4850bbf16b4827e92922 (127.15.47.129). New cstate: current_term: 1 leader_uuid: "7575b61d310d4850bbf16b4827e92922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:41.754158 16046 tablet.cc:2369] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:41.754885 15914 tablet.cc:2369] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:41.759189 15781 tablet.cc:2369] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:41.778548 16070 raft_consensus.cc:482] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.778786 16070 raft_consensus.cc:504] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.780543 16070 leader_election.cc:288] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.787204 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d451ab3b430dab914b22d4dbb83b" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:41.787457 15999 raft_consensus.cc:2454] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 0.
I20240502 21:25:41.787431 15735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d451ab3b430dab914b22d4dbb83b" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7575b61d310d4850bbf16b4827e92922" is_pre_election: true
I20240502 21:25:41.787917 15735 raft_consensus.cc:2454] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 0.
I20240502 21:25:41.787943 15802 leader_election.cc:302] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters:
I20240502 21:25:41.788426 16070 raft_consensus.cc:2781] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.788594 16070 raft_consensus.cc:482] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.788705 16070 raft_consensus.cc:3037] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.790699 16070 raft_consensus.cc:504] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.791308 16070 leader_election.cc:288] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.791766 15735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d451ab3b430dab914b22d4dbb83b" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7575b61d310d4850bbf16b4827e92922"
I20240502 21:25:41.791810 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d451ab3b430dab914b22d4dbb83b" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:41.791963 15735 raft_consensus.cc:3037] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.791998 15999 raft_consensus.cc:3037] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.793896 15999 raft_consensus.cc:2454] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 1.
I20240502 21:25:41.793990 15735 raft_consensus.cc:2454] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 1.
I20240502 21:25:41.794297 15802 leader_election.cc:302] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters:
I20240502 21:25:41.794610 16070 raft_consensus.cc:2781] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.795035 16070 raft_consensus.cc:686] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1df45829447d4299992c186d49363fb4, State: Running, Role: LEADER
I20240502 21:25:41.795564 16070 consensus_queue.cc:237] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } }
I20240502 21:25:41.800226 15592 catalog_manager.cc:5484] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df45829447d4299992c186d49363fb4 (127.15.47.130). New cstate: current_term: 1 leader_uuid: "1df45829447d4299992c186d49363fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:41.935806 16073 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.936021 16073 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.936725 16073 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.937297 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4" is_pre_election: true
I20240502 21:25:41.937371 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:41.937589 15999 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 0.
I20240502 21:25:41.937623 15867 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 0.
I20240502 21:25:41.938033 15669 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e53385ddc904372a6685f865d7c9e28, 7575b61d310d4850bbf16b4827e92922; no voters:
I20240502 21:25:41.938295 16073 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.938380 16073 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.938501 16073 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.940574 16073 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.941130 16073 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 election: Requested vote from peers 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:41.941597 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4"
I20240502 21:25:41.941682 15999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "7575b61d310d4850bbf16b4827e92922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:41.941859 15867 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.941890 15999 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.943835 15999 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 1.
I20240502 21:25:41.943853 15867 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7575b61d310d4850bbf16b4827e92922 in term 1.
I20240502 21:25:41.944226 15669 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e53385ddc904372a6685f865d7c9e28, 7575b61d310d4850bbf16b4827e92922; no voters:
I20240502 21:25:41.944515 16073 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.944686 16073 raft_consensus.cc:686] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [term 1 LEADER]: Becoming Leader. State: Replica: 7575b61d310d4850bbf16b4827e92922, State: Running, Role: LEADER
I20240502 21:25:41.944995 16073 consensus_queue.cc:237] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:41.947821 15592 catalog_manager.cc:5484] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7575b61d310d4850bbf16b4827e92922 (127.15.47.129). New cstate: current_term: 1 leader_uuid: "7575b61d310d4850bbf16b4827e92922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:42.027878 15550 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:25:42.031893 15550 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 7575b61d310d4850bbf16b4827e92922 to finish bootstrapping
I20240502 21:25:42.050334 15550 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 1df45829447d4299992c186d49363fb4 to finish bootstrapping
I20240502 21:25:42.058707 15550 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5e53385ddc904372a6685f865d7c9e28 to finish bootstrapping
I20240502 21:25:42.070940 16090 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.15.47.190:39895/dump-entities
http://127.15.47.190:39895/threadz?group=all
http://127.15.47.130:44187/mem-trackers
http://127.15.47.130:44187/scans
http://127.15.47.130:44187/transactions?tablet_id=23c3098e6fc94593a4adbd5b89076b2c
http://127.15.47.130:44187/stacks
http://127.15.47.131:37807/stacks
http://127.15.47.131:37807/threadz?group=all
http://127.15.47.129:37047/tablets
http://127.15.47.190:39895/tables
http://127.15.47.131:37807/scans
http://127.15.47.130:44187/threadz
http://127.15.47.190:39895/ipki-ca-cert-pem
http://127.15.47.131:37807/startup
http://127.15.47.129:37047/metrics
http://127.15.47.130:44187/tablets
http://127.15.47.129:37047/threadz?group=all
http://127.15.47.130:44187/startup
http://127.15.47.129:37047/maintenance-manager
http://127.15.47.131:37807/maintenance-manager
http://127.15.47.129:37047/transactions?tablet_id=23c3098e6fc94593a4adbd5b89076b2c
http://127.15.47.130:44187/maintenance-manager
http://127.15.47.131:37807/transactions?tablet_id=23c3098e6fc94593a4adbd5b89076b2c
http://127.15.47.129:37047/mem-trackers
http://127.15.47.131:37807/metrics
http://127.15.47.190:39895/masters
http://127.15.47.190:39895/tablet-servers
http://127.15.47.129:37047/stacks
http://127.15.47.190:39895/threadz
http://127.15.47.129:37047/scans
http://127.15.47.130:44187/metrics
http://127.15.47.190:39895/ipki-ca-cert-der
http://127.15.47.129:37047/startup
http://127.15.47.131:37807/tablets
http://127.15.47.130:44187/threadz?group=all
http://127.15.47.190:39895/startup
http://127.15.47.131:37807/mem-trackers
http://127.15.47.190:39895/ipki-ca-cert
http://127.15.47.190:39895/stacks
http://127.15.47.131:37807/threadz
http://127.15.47.190:39895/mem-trackers
http://127.15.47.129:37047/threadz
http://127.15.47.190:39895/metrics
I20240502 21:25:42.077666 15550 linked_list-test-util.h:401] Running for: 30.000s
I20240502 21:25:42.077840 15550 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 21:25:42.077953 15550 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 21:25:42.095304 16091 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/16058/stat: No such file or directory (error 2)
I20240502 21:25:42.103875 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:42.135190 16073 consensus_queue.cc:1035] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.144160 15999 raft_consensus.cc:1261] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Refusing update from remote peer 7575b61d310d4850bbf16b4827e92922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:42.145337 16073 consensus_queue.cc:1035] T 23c3098e6fc94593a4adbd5b89076b2c P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.147569 15998 raft_consensus.cc:1261] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Refusing update from remote peer 7575b61d310d4850bbf16b4827e92922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:42.148572 16073 consensus_queue.cc:1035] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.152211 15866 raft_consensus.cc:1261] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7575b61d310d4850bbf16b4827e92922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:42.153259 16073 consensus_queue.cc:1035] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 7575b61d310d4850bbf16b4827e92922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.163911 15735 raft_consensus.cc:1261] T 26d7d451ab3b430dab914b22d4dbb83b P 7575b61d310d4850bbf16b4827e92922 [term 1 FOLLOWER]: Refusing update from remote peer 1df45829447d4299992c186d49363fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:42.165251 16086 consensus_queue.cc:1035] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.165872 15997 raft_consensus.cc:1261] T 26d7d451ab3b430dab914b22d4dbb83b P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Refusing update from remote peer 1df45829447d4299992c186d49363fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:42.168607 16086 consensus_queue.cc:1035] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.221532 16099 mvcc.cc:205] Tried to move back new op lower bound from 7023350342268375040 to 7023350340798935040. Current Snapshot: MvccSnapshot[applied={T|T < 7023350342268375040}]
I20240502 21:25:42.225924 16097 mvcc.cc:205] Tried to move back new op lower bound from 7023350342268375040 to 7023350340798935040. Current Snapshot: MvccSnapshot[applied={T|T < 7023350342268375040}]
I20240502 21:25:42.274642 16105 mvcc.cc:205] Tried to move back new op lower bound from 7023350342221754368 to 7023350341408837632. Current Snapshot: MvccSnapshot[applied={T|T < 7023350342221754368}]
I20240502 21:25:43.947693 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:44.895259 16136 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:45.728546 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:47.631000 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:47.771682 15781 maintenance_manager.cc:419] P 7575b61d310d4850bbf16b4827e92922: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:25:47.779136 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:25:47.779501 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:25:50.092486 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 2.312s user 0.547s sys 0.101s Metrics: {"bytes_written":2760,"cfile_init":1,"compiler_manager_pool.queue_time_us":385702,"compiler_manager_pool.run_cpu_time_us":671530,"compiler_manager_pool.run_wall_time_us":1934861,"delete_count":0,"dirs.queue_time_us":6762,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":2161,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13257,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7831,"thread_start_us":7242,"threads_started":2,"update_count":401}
I20240502 21:25:50.119882 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.022240
I20240502 21:25:50.282282 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 2.502s user 0.575s sys 0.089s Metrics: {"bytes_written":2967,"cfile_init":1,"compiler_manager_pool.queue_time_us":551920,"compiler_manager_pool.run_cpu_time_us":663121,"compiler_manager_pool.run_wall_time_us":2356528,"delete_count":0,"dirs.queue_time_us":27554,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":7604,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2261,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7863,"thread_start_us":107983,"threads_started":2,"update_count":433}
I20240502 21:25:50.285131 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 136067 bytes on disk
I20240502 21:25:50.285424 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.163s user 0.066s sys 0.019s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3663,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":18725,"dirs.run_cpu_time_us":318,"dirs.run_wall_time_us":3954,"lbm_read_time_us":681,"lbm_reads_lt_1ms":24,"lbm_write_time_us":5436,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"thread_start_us":24379,"threads_started":2,"update_count":401}
I20240502 21:25:50.286406 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 135728 bytes on disk
I20240502 21:25:50.287256 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:25:50.292259 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 21:25:50.293109 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.023802
I20240502 21:25:50.296744 15671 maintenance_manager.cc:643] P 7575b61d310d4850bbf16b4827e92922: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 2.524s user 0.575s sys 0.084s Metrics: {"bytes_written":2760,"cfile_init":1,"compiler_manager_pool.queue_time_us":372813,"compiler_manager_pool.run_cpu_time_us":673282,"compiler_manager_pool.run_wall_time_us":2383226,"delete_count":0,"dirs.queue_time_us":4897,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":5508,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2363,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7831,"thread_start_us":13020,"threads_started":2,"update_count":401}
I20240502 21:25:50.298951 15781 maintenance_manager.cc:419] P 7575b61d310d4850bbf16b4827e92922: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 135728 bytes on disk
I20240502 21:25:50.300176 15671 maintenance_manager.cc:643] P 7575b61d310d4850bbf16b4827e92922: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":299,"lbm_reads_lt_1ms":4}
I20240502 21:25:50.301066 15781 maintenance_manager.cc:419] P 7575b61d310d4850bbf16b4827e92922: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.022240
I20240502 21:25:50.436035 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.143s user 0.055s sys 0.027s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3905,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":8039,"dirs.run_cpu_time_us":316,"dirs.run_wall_time_us":1670,"lbm_read_time_us":405,"lbm_reads_lt_1ms":20,"lbm_write_time_us":326,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":15840,"threads_started":2,"update_count":433}
I20240502 21:25:50.445343 15671 maintenance_manager.cc:643] P 7575b61d310d4850bbf16b4827e92922: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.139s user 0.050s sys 0.019s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3663,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1151,"lbm_read_time_us":4047,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":19,"lbm_write_time_us":514,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":802,"threads_started":2,"update_count":401}
I20240502 21:25:50.734889 16136 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:51.514292 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:52.086314 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350382891790336 Inserted count: 19000
I20240502 21:25:53.112593 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 21:25:53.233096 16136 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7f0fbf567dcd google::LogMessage::Fail() at ??:0
@ 0x7f0fbf56bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0fbf5677cc google::LogMessage::Flush() at ??:0
@ 0x7f0fbf568f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0fc2190c80 kudu::operator-() at ??:0
@ 0x7f0fcee1114a kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7f0fceddb5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7f0fceddb336 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7f0fced882d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7f0fd1855e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7f0fd18649fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
@ 0x7f0fd1864862 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0fcb4a404a std::function<>::operator()() at ??:0
@ 0x7f0fcb49c806 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7f0fcb497811 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7f0fcb495947 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7f0fcb4d30a9 handle_request at ??:0
@ 0x7f0fcb4d578b worker_thread at ??:0
@ 0x7f0fc55dc6db start_thread at ??:0
@ 0x7f0fbe41a71f clone at ??:0
W20240502 21:25:53.491565 16052 meta_cache.cc:302] tablet ce42e0503d2c4b08a3a825f2d1e6fd01: replica 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827) has failed: Network error: recv got EOF from 127.15.47.129:38827 (error 108)
W20240502 21:25:53.494711 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.510581 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.530894 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.541155 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.565799 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.578419 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.610627 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.624650 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.625576 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.627892 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.628161 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.634341 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.647980 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:53.660434 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:53.665865 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.667886 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.674155 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.676144 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.684150 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.687286 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.700209 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.713953 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.724417 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.744328 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.744386 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.755153 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.762223 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.790843 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.800197 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.807335 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.817467 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.818723 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.857082 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.867632 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.874979 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.878966 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.890311 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.904341 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.926324 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.953765 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.964706 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:53.995381 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.003391 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.012257 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.031306 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.088632 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.091784 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.093173 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
I20240502 21:25:54.111828 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:25:54.117775 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.127583 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.182338 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.203917 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.216668 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.222113 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.240934 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.284910 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.318979 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.319311 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.331666 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.358454 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.396327 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.438050 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.443776 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.485603 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.494733 16052 meta_cache.cc:302] tablet 23c3098e6fc94593a4adbd5b89076b2c: replica 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827) has failed: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:54.522056 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.563994 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.577795 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.582127 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.631103 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.658754 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.666990 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.705950 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.726074 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
I20240502 21:25:54.728848 16226 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.729058 16226 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.730667 16226 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433)
W20240502 21:25:54.731808 15934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:54.733726 15934 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.736737 16225 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.736974 16225 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.738003 16225 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:54.738761 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:54.739061 15997 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 1.
W20240502 21:25:54.739578 15803 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.739956 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "5e53385ddc904372a6685f865d7c9e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4" is_pre_election: true
I20240502 21:25:54.740119 15802 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.740188 15865 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e53385ddc904372a6685f865d7c9e28 in term 1.
I20240502 21:25:54.740485 16225 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:54.740618 16225 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.740721 16225 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:54.740761 15933 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.741128 16226 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:54.741328 16226 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.741449 16226 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:54.744091 16225 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.744167 16226 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.745265 16226 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433)
I20240502 21:25:54.746615 16226 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
W20240502 21:25:54.746873 15803 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.746922 16226 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.747316 16225 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:54.747560 16225 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.747936 16226 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433)
W20240502 21:25:54.748708 15934 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.748651 16225 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
W20240502 21:25:54.750232 15934 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:54.750942 15803 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.751013 16225 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:54.752043 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:54.752314 15998 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 1.
I20240502 21:25:54.751973 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:54.752404 15865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "5e53385ddc904372a6685f865d7c9e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4"
I20240502 21:25:54.752698 15997 raft_consensus.cc:2379] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df45829447d4299992c186d49363fb4 in current term 2: Already voted for candidate 5e53385ddc904372a6685f865d7c9e28 in this term.
I20240502 21:25:54.752736 15865 raft_consensus.cc:2379] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e53385ddc904372a6685f865d7c9e28 in current term 2: Already voted for candidate 1df45829447d4299992c186d49363fb4 in this term.
I20240502 21:25:54.753443 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "5e53385ddc904372a6685f865d7c9e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4" is_pre_election: true
I20240502 21:25:54.753667 15866 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e53385ddc904372a6685f865d7c9e28 in term 1.
I20240502 21:25:54.754081 15802 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.754122 15933 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e53385ddc904372a6685f865d7c9e28; no voters: 1df45829447d4299992c186d49363fb4, 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.754554 15933 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.754551 15802 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df45829447d4299992c186d49363fb4; no voters: 5e53385ddc904372a6685f865d7c9e28, 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.755007 16231 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:54.755118 16231 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.755275 16231 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:54.755785 16225 raft_consensus.cc:2726] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:25:54.756060 16226 raft_consensus.cc:2726] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:25:54.756664 16226 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:54.756886 16226 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7575b61d310d4850bbf16b4827e92922)
I20240502 21:25:54.757229 16226 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:54.758533 16231 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.759876 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:54.760207 16226 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:54.760710 15997 raft_consensus.cc:2379] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df45829447d4299992c186d49363fb4 in current term 2: Already voted for candidate 5e53385ddc904372a6685f865d7c9e28 in this term.
I20240502 21:25:54.761142 16226 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 1df45829447d4299992c186d49363fb4 (127.15.47.130:35433)
W20240502 21:25:54.761524 15803 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.761529 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "5e53385ddc904372a6685f865d7c9e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "1df45829447d4299992c186d49363fb4"
I20240502 21:25:54.761837 15866 raft_consensus.cc:2379] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e53385ddc904372a6685f865d7c9e28 in current term 2: Already voted for candidate 1df45829447d4299992c186d49363fb4 in this term.
I20240502 21:25:54.762173 15802 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df45829447d4299992c186d49363fb4; no voters: 5e53385ddc904372a6685f865d7c9e28, 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.762439 16225 raft_consensus.cc:2726] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:25:54.762707 16231 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 2 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
W20240502 21:25:54.762893 15934 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:54.763011 15934 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e53385ddc904372a6685f865d7c9e28; no voters: 1df45829447d4299992c186d49363fb4, 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:54.763245 16226 raft_consensus.cc:2726] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 21:25:54.776013 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.803932 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.854000 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.879480 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.933861 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.954918 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:54.963460 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.004853 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.044744 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.101011 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.113701 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.122157 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.170349 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
I20240502 21:25:55.188130 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:25:55.217825 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.278156 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.288208 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.344909 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.397856 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.458580 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.464515 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.467758 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.523804 16052 meta_cache.cc:302] tablet ce42e0503d2c4b08a3a825f2d1e6fd01: replica 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827) has failed: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:55.526710 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.588588 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.590924 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.653139 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.655833 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
I20240502 21:25:55.771009 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:25:55.785059 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.787278 15959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.787317 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.851045 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.923130 15827 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.923130 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:55.991740 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.009204 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:56.066819 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.131816 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.137976 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.191603 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.201984 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.281311 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.346704 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.354243 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.410836 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.421166 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
I20240502 21:25:56.450562 16231 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:56.450863 16231 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.451572 16231 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:56.452350 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:56.452625 15997 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 2.
I20240502 21:25:56.453195 15802 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters:
W20240502 21:25:56.453460 15803 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:56.453524 16231 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:25:56.453634 16231 raft_consensus.cc:482] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:56.453747 16231 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:25:56.456149 16231 raft_consensus.cc:504] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.456881 16231 leader_election.cc:288] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:56.457597 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce42e0503d2c4b08a3a825f2d1e6fd01" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 429 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:56.457911 15997 raft_consensus.cc:3037] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Advancing to term 3
W20240502 21:25:56.458168 15803 leader_election.cc:334] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:56.461885 15997 raft_consensus.cc:2454] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 3.
I20240502 21:25:56.462494 15802 leader_election.cc:302] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:56.462872 16231 raft_consensus.cc:2781] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:25:56.463328 16231 raft_consensus.cc:686] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [term 3 LEADER]: Becoming Leader. State: Replica: 1df45829447d4299992c186d49363fb4, State: Running, Role: LEADER
I20240502 21:25:56.463655 16231 consensus_queue.cc:237] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 429, Committed index: 429, Last appended: 1.429, Last appended by leader: 429, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.468050 15589 catalog_manager.cc:5484] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 reported cstate change: term changed from 1 to 3, leader changed from 7575b61d310d4850bbf16b4827e92922 (127.15.47.129) to 1df45829447d4299992c186d49363fb4 (127.15.47.130). New cstate: current_term: 3 leader_uuid: "1df45829447d4299992c186d49363fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:56.502149 15997 raft_consensus.cc:1261] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 5e53385ddc904372a6685f865d7c9e28 [term 3 FOLLOWER]: Refusing update from remote peer 1df45829447d4299992c186d49363fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 429. Preceding OpId from leader: term: 3 index: 431. (index mismatch)
I20240502 21:25:56.502875 16231 consensus_queue.cc:1035] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 430, Last known committed idx: 429, Time since last communication: 0.000s
W20240502 21:25:56.503298 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:56.540257 16052 meta_cache.cc:302] tablet 23c3098e6fc94593a4adbd5b89076b2c: replica 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827) has failed: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
W20240502 21:25:56.546574 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.553040 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.568022 15826 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41864: Illegal state: replica 1df45829447d4299992c186d49363fb4 is not leader of this config: current role FOLLOWER
W20240502 21:25:56.579643 15958 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33314: Illegal state: replica 5e53385ddc904372a6685f865d7c9e28 is not leader of this config: current role FOLLOWER
I20240502 21:25:56.581887 16248 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:56.582060 16248 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.582844 16248 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:56.583415 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28" is_pre_election: true
I20240502 21:25:56.583703 15997 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 2.
I20240502 21:25:56.584169 15802 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters:
I20240502 21:25:56.584532 16248 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 21:25:56.584580 15803 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:56.584705 16248 raft_consensus.cc:482] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:56.584838 16248 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:25:56.588367 16248 raft_consensus.cc:504] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.589301 16248 leader_election.cc:288] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: Requested vote from peers 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827), 5e53385ddc904372a6685f865d7c9e28 (127.15.47.131:40499)
I20240502 21:25:56.589941 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c3098e6fc94593a4adbd5b89076b2c" candidate_uuid: "1df45829447d4299992c186d49363fb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 430 } } ignore_live_leader: false dest_uuid: "5e53385ddc904372a6685f865d7c9e28"
I20240502 21:25:56.590229 15997 raft_consensus.cc:3037] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 2 FOLLOWER]: Advancing to term 3
W20240502 21:25:56.590675 15803 leader_election.cc:334] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111)
I20240502 21:25:56.593834 15997 raft_consensus.cc:2454] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df45829447d4299992c186d49363fb4 in term 3.
I20240502 21:25:56.594345 15802 leader_election.cc:302] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df45829447d4299992c186d49363fb4, 5e53385ddc904372a6685f865d7c9e28; no voters: 7575b61d310d4850bbf16b4827e92922
I20240502 21:25:56.594671 16248 raft_consensus.cc:2781] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:25:56.594884 16248 raft_consensus.cc:686] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [term 3 LEADER]: Becoming Leader. State: Replica: 1df45829447d4299992c186d49363fb4, State: Running, Role: LEADER
I20240502 21:25:56.595305 16248 consensus_queue.cc:237] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 1.430, Last appended by leader: 430, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } }
I20240502 21:25:56.598589 15589 catalog_manager.cc:5484] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 reported cstate change: term changed from 1 to 3, leader changed from 7575b61d310d4850bbf16b4827e92922 (127.15.47.129) to 1df45829447d4299992c186d49363fb4 (127.15.47.130). New cstate: current_term: 3 leader_uuid: "1df45829447d4299992c186d49363fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7575b61d310d4850bbf16b4827e92922" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 38827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df45829447d4299992c186d49363fb4" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:56.607815 15997 raft_consensus.cc:1261] T 23c3098e6fc94593a4adbd5b89076b2c P 5e53385ddc904372a6685f865d7c9e28 [term 3 FOLLOWER]: Refusing update from remote peer 1df45829447d4299992c186d49363fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 430. Preceding OpId from leader: term: 3 index: 432. (index mismatch)
W20240502 21:25:56.608016 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.608561 16248 consensus_queue.cc:1035] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e53385ddc904372a6685f865d7c9e28" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I20240502 21:25:56.860996 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:57.453708 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:25:57.556766 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:25:57.759269 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:25:58.229156 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.775s user 0.485s sys 0.091s Metrics: {"bytes_written":11260,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13320,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":8573,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2313,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8128,"spinlock_wait_cycles":15104,"thread_start_us":13451,"threads_started":1,"update_count":1738}
I20240502 21:25:58.230453 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014692
W20240502 21:25:58.381289 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:58.406747 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.850s user 0.518s sys 0.063s Metrics: {"bytes_written":11924,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":16494,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":5755,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2133,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8293,"spinlock_wait_cycles":1408,"thread_start_us":16389,"threads_started":1,"update_count":1837}
I20240502 21:25:58.408390 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014819
W20240502 21:25:58.676889 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240502 21:25:59.041539 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.807s user 0.638s sys 0.027s Metrics: {"bytes_written":292009,"cfile_cache_hit":14,"cfile_cache_hit_bytes":93108,"cfile_cache_miss":62,"cfile_cache_miss_bytes":971296,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":10527,"dirs.run_cpu_time_us":925,"dirs.run_wall_time_us":15830,"drs_written":1,"lbm_read_time_us":3374,"lbm_reads_lt_1ms":102,"lbm_write_time_us":4561,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":50,"mutex_wait_us":25,"num_input_rowsets":2,"peak_mem_usage":525516,"rows_written":15991,"spinlock_wait_cycles":2816,"thread_start_us":24050,"threads_started":3}
I20240502 21:25:59.042771 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 292418 bytes on disk
I20240502 21:25:59.043421 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240502 21:25:59.117684 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:59.214602 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:59.408185 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 1.000s user 0.700s sys 0.020s Metrics: {"bytes_written":398,"cfile_cache_hit":15,"cfile_cache_hit_bytes":141493,"cfile_cache_miss":63,"cfile_cache_miss_bytes":931905,"cfile_init":10,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":6223,"dirs.run_cpu_time_us":1520,"dirs.run_wall_time_us":8889,"drs_written":1,"lbm_read_time_us":3303,"lbm_reads_lt_1ms":103,"lbm_write_time_us":6958,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":529365,"reinsert_count":0,"rows_written":16124,"thread_start_us":8658,"threads_started":6,"update_count":34}
I20240502 21:25:59.409864 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 294659 bytes on disk
I20240502 21:25:59.411283 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 21:25:59.415220 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.001573
I20240502 21:25:59.528107 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.112s user 0.079s sys 0.015s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":506,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":93,"lbm_read_time_us":458,"lbm_reads_lt_1ms":20,"lbm_write_time_us":329,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":34}
I20240502 21:25:59.960215 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:00.832253 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:00.950102 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:26:01.623809 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:26:01.915333 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:26:02.095814 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350423916462080 Inserted count: 32500
I20240502 21:26:02.285178 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:02.992869 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:03.290247 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:26:03.788395 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:26:03.791309 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:26:03.815021 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
W20240502 21:26:04.224509 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:26:04.651664 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:26:04.659865 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.845s user 0.509s sys 0.072s Metrics: {"bytes_written":11960,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15245,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":141394,"drs_written":1,"lbm_read_time_us":1047,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2643,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8225,"thread_start_us":15094,"threads_started":1,"update_count":1842}
I20240502 21:26:04.660593 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.869s user 0.550s sys 0.056s Metrics: {"bytes_written":11965,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6882,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":182267,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2531,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8057,"thread_start_us":7010,"threads_started":1,"update_count":1842}
I20240502 21:26:04.662290 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014702
I20240502 21:26:04.662562 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014700
I20240502 21:26:04.844415 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:05.731184 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:05.765149 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 1.102s user 0.923s sys 0.029s Metrics: {"bytes_written":406,"cfile_cache_hit":12,"cfile_cache_hit_bytes":49103,"cfile_cache_miss":80,"cfile_cache_miss_bytes":1591087,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":27015,"dirs.run_cpu_time_us":978,"dirs.run_wall_time_us":19792,"drs_written":1,"lbm_read_time_us":4334,"lbm_reads_lt_1ms":116,"lbm_write_time_us":4953,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":800410,"reinsert_count":0,"rows_written":24216,"thread_start_us":41177,"threads_started":3,"update_count":35}
I20240502 21:26:05.766525 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 450168 bytes on disk
I20240502 21:26:05.767220 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 21:26:05.768126 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.001053
W20240502 21:26:05.864935 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:26:05.871871 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.103s user 0.094s sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":518,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":79,"lbm_read_time_us":370,"lbm_reads_lt_1ms":20,"lbm_write_time_us":345,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":35}
I20240502 21:26:05.995934 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 1.333s user 1.231s sys 0.028s Metrics: {"bytes_written":446959,"cfile_cache_hit":15,"cfile_cache_hit_bytes":233862,"cfile_cache_miss":81,"cfile_cache_miss_bytes":1401864,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":11744,"dirs.run_cpu_time_us":1380,"dirs.run_wall_time_us":22571,"drs_written":1,"lbm_read_time_us":4832,"lbm_reads_lt_1ms":121,"lbm_write_time_us":5160,"lbm_writes_lt_1ms":64,"mutex_wait_us":58,"num_input_rowsets":2,"peak_mem_usage":799867,"rows_written":24181,"spinlock_wait_cycles":7552,"thread_start_us":23208,"threads_started":3}
I20240502 21:26:05.997906 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 449738 bytes on disk
I20240502 21:26:05.998768 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:26:06.723318 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:26:07.227635 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:26:07.481556 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:08.341521 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:08.379757 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(ce42e0503d2c4b08a3a825f2d1e6fd01): perf score=1.000000
W20240502 21:26:08.473817 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:26:08.505488 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(23c3098e6fc94593a4adbd5b89076b2c): perf score=1.000000
W20240502 21:26:08.880898 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:26:08.977424 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(ce42e0503d2c4b08a3a825f2d1e6fd01) complete. Timing: real 0.597s user 0.430s sys 0.086s Metrics: {"bytes_written":143193,"cfile_init":1,"dirs.queue_time_us":6370,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":4611,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2335,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7626,"thread_start_us":6520,"threads_started":1}
I20240502 21:26:08.978713 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(ce42e0503d2c4b08a3a825f2d1e6fd01): 142264 bytes on disk
I20240502 21:26:08.979527 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(ce42e0503d2c4b08a3a825f2d1e6fd01) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 21:26:08.980512 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(23c3098e6fc94593a4adbd5b89076b2c): perf score=1.000000
W20240502 21:26:09.130558 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:26:09.286218 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.780s user 0.546s sys 0.049s Metrics: {"bytes_written":1212,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11824,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":7809,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2495,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7584,"thread_start_us":11734,"threads_started":1,"update_count":162}
I20240502 21:26:09.287463 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(23c3098e6fc94593a4adbd5b89076b2c): 141475 bytes on disk
I20240502 21:26:09.300750 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 21:26:09.303396 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(ce42e0503d2c4b08a3a825f2d1e6fd01): perf score=1.000000
I20240502 21:26:09.663609 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.680s user 0.457s sys 0.011s Metrics: {"bytes_written":1966,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12868,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":2319,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2647,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7701,"spinlock_wait_cycles":23808,"thread_start_us":13003,"threads_started":1,"update_count":279}
I20240502 21:26:09.665194 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling MajorDeltaCompactionOp(23c3098e6fc94593a4adbd5b89076b2c): perf score=0.015886
I20240502 21:26:09.732831 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: MajorDeltaCompactionOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.066s user 0.042s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2668,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1500,"dirs.run_cpu_time_us":314,"dirs.run_wall_time_us":2341,"lbm_read_time_us":469,"lbm_reads_lt_1ms":24,"lbm_write_time_us":389,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":12499,"threads_started":1,"update_count":279}
I20240502 21:26:09.733845 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(23c3098e6fc94593a4adbd5b89076b2c): 142789 bytes on disk
I20240502 21:26:09.734503 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 21:26:09.927331 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:26:09.956388 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(ce42e0503d2c4b08a3a825f2d1e6fd01) complete. Timing: real 0.650s user 0.439s sys 0.054s Metrics: {"bytes_written":145747,"cfile_init":1,"dirs.queue_time_us":4630,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2461,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2297,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7828,"thread_start_us":4514,"threads_started":1}
I20240502 21:26:09.957547 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(ce42e0503d2c4b08a3a825f2d1e6fd01): 146940 bytes on disk
I20240502 21:26:09.958268 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(ce42e0503d2c4b08a3a825f2d1e6fd01) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 21:26:09.958988 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling MajorDeltaCompactionOp(23c3098e6fc94593a4adbd5b89076b2c): perf score=0.009958
I20240502 21:26:10.002216 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: MajorDeltaCompactionOp(23c3098e6fc94593a4adbd5b89076b2c) complete. Timing: real 0.043s user 0.037s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1711,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":761,"lbm_read_time_us":373,"lbm_reads_lt_1ms":20,"lbm_write_time_us":391,"lbm_writes_lt_1ms":8,"mutex_wait_us":158,"reinsert_count":0,"thread_start_us":402,"threads_started":1,"update_count":162}
I20240502 21:26:10.237592 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:26:10.254333 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.000000
I20240502 21:26:10.573406 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:10.852028 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.597s user 0.544s sys 0.012s Metrics: {"bytes_written":7784,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6347,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":3472,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2476,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7996,"thread_start_us":6549,"threads_started":1,"update_count":1189}
I20240502 21:26:10.853453 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014572
I20240502 21:26:10.912454 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: FlushMRSOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.674s user 0.429s sys 0.021s Metrics: {"bytes_written":145531,"cfile_init":1,"dirs.queue_time_us":1319,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1013,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2102,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7994,"spinlock_wait_cycles":256768,"thread_start_us":1193,"threads_started":1}
I20240502 21:26:10.913727 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 141904 bytes on disk
I20240502 21:26:10.916954 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 21:26:10.918314 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=1.014570
W20240502 21:26:10.941911 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:26:11.608311 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:26:11.716624 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:12.123525 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350464925892608 Inserted count: 51250
I20240502 21:26:12.124372 15550 linked_list-test-util.h:421] Finished inserting list. Added 51250 in chain
I20240502 21:26:12.124542 15550 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 21:26:12.124666 15550 linked_list-test-util.h:424] 0: -8291122034670043136
I20240502 21:26:12.124823 15550 linked_list-test-util.h:424] 1: 6638356274953322497
I20240502 21:26:12.124987 15550 linked_list-test-util.h:424] 2: 3120809035890425858
I20240502 21:26:12.125101 15550 linked_list-test-util.h:424] 3: -396597465684049917
I20240502 21:26:12.125223 15550 linked_list-test-util.h:424] 4: 6241758809269272580
I20240502 21:26:12.125324 15550 linked_list-test-util.h:424] 5: 2724493045183086597
I20240502 21:26:12.125435 15550 linked_list-test-util.h:424] 6: -792913456391389178
I20240502 21:26:12.125531 15550 linked_list-test-util.h:424] 7: -4310460695454285817
I20240502 21:26:12.125952 15550 linked_list-test-util.h:424] 8: 2327895579499036680
I20240502 21:26:12.126068 15550 linked_list-test-util.h:424] 9: -1189510922075439095
I20240502 21:26:12.126186 15550 linked_list-test-util.h:424] 10: -4706776686161625078
I20240502 21:26:12.126281 15550 linked_list-test-util.h:424] 11: -8224323925224521717
I20240502 21:26:12.136384 15550 linked_list-test-util.h:424] 12: -1585826912782778356
I20240502 21:26:12.136587 15550 linked_list-test-util.h:424] 13: -5103374151845674995
I20240502 21:26:12.136713 15550 linked_list-test-util.h:424] 14: -8620639915931860978
I20240502 21:26:12.136808 15550 linked_list-test-util.h:424] 15: 6308556918714793999
I20240502 21:26:12.136933 15550 linked_list-test-util.h:424] 16: -5499690142553014256
I20240502 21:26:12.137029 15550 linked_list-test-util.h:424] 17: -9017237381615910895
I20240502 21:26:12.137149 15550 linked_list-test-util.h:424] 18: 5912240928007454738
I20240502 21:26:12.137243 15550 linked_list-test-util.h:424] 19: 2394834426432978963
I20240502 21:26:12.137355 15550 linked_list-test-util.h:424] 20: 9033190701386301460
I20240502 21:26:12.137449 15550 linked_list-test-util.h:424] 21: 5515784199811825685
I20240502 21:26:12.137555 15550 linked_list-test-util.h:424] 22: 1998236960748929046
I20240502 21:26:12.137648 15550 linked_list-test-util.h:424] 23: -1519028803337256937
I20240502 21:26:12.137755 15550 linked_list-test-util.h:424] 24: 5119186734127775768
I20240502 21:26:12.137851 15550 linked_list-test-util.h:424] 25: 1601920970041589785
I20240502 21:26:12.137969 15550 linked_list-test-util.h:424] 26: -1915626269021306854
I20240502 21:26:12.138075 15550 linked_list-test-util.h:424] 27: 4722870743420436507
I20240502 21:26:12.138185 15550 linked_list-test-util.h:424] 28: 1205323504357539868
I20240502 21:26:12.138271 15550 linked_list-test-util.h:424] 29: -2311942259728646115
I20240502 21:26:12.138372 15550 linked_list-test-util.h:424] 30: -5829489498791542754
I20240502 21:26:12.138463 15550 linked_list-test-util.h:424] 31: 809007513650200607
I20240502 21:26:12.138568 15550 linked_list-test-util.h:424] 32: -2708539725412696032
I20240502 21:26:12.138659 15550 linked_list-test-util.h:424] 33: -6225805489498882015
I20240502 21:26:12.138765 15550 linked_list-test-util.h:424] 34: 8703532082636193826
I20240502 21:26:12.138877 15550 linked_list-test-util.h:424] 35: -3104855716120035293
I20240502 21:26:12.139014 15550 linked_list-test-util.h:424] 36: -6622121480206221276
I20240502 21:26:12.139130 15550 linked_list-test-util.h:424] 37: 8306934616952143909
I20240502 21:26:12.139250 15550 linked_list-test-util.h:424] 38: 4789668852865957926
I20240502 21:26:12.139339 15550 linked_list-test-util.h:424] 39: -7018859683378560985
I20240502 21:26:12.139446 15550 linked_list-test-util.h:424] 40: 7910618626244804648
I20240502 21:26:12.139536 15550 linked_list-test-util.h:424] 41: 4393071387181908009
I20240502 21:26:12.139645 15550 linked_list-test-util.h:424] 42: 875805623095722026
I20240502 21:26:12.139734 15550 linked_list-test-util.h:424] 43: 7514021160560754731
I20240502 21:26:12.139844 15550 linked_list-test-util.h:424] 44: 3996755396474568748
I20240502 21:26:12.139941 15550 linked_list-test-util.h:424] 45: 479208157411672109
I20240502 21:26:12.140048 15550 linked_list-test-util.h:424] 46: -3038198344162803666
I20240502 21:26:12.140139 15550 linked_list-test-util.h:424] 47: 3600157930790518831
I20240502 21:26:12.140242 15550 linked_list-test-util.h:424] 48: 82892166704332848
I20240502 21:26:12.140350 15550 linked_list-test-util.h:424] 49: -3434514334870142927
W20240502 21:26:12.253202 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:26:12.300170 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685152073191 usec, L: 1 Expected Rows: 19000
I20240502 21:26:12.436005 15550 linked_list-test-util.h:528] Calling callback on tserver 1df45829447d4299992c186d49363fb4
I20240502 21:26:12.895670 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 2.042s user 1.514s sys 0.026s Metrics: {"bytes_written":392,"cfile_cache_hit":12,"cfile_cache_hit_bytes":48058,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2152400,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":16621,"dirs.run_cpu_time_us":2411,"dirs.run_wall_time_us":37940,"drs_written":1,"lbm_read_time_us":6648,"lbm_reads_lt_1ms":130,"lbm_write_time_us":6959,"lbm_writes_lt_1ms":80,"mutex_wait_us":176,"num_input_rowsets":2,"peak_mem_usage":1067140,"reinsert_count":0,"rows_written":32177,"thread_start_us":26828,"threads_started":5,"update_count":33}
I20240502 21:26:12.897141 16046 maintenance_manager.cc:419] P 5e53385ddc904372a6685f865d7c9e28: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 605553 bytes on disk
I20240502 21:26:12.897851 15935 maintenance_manager.cc:643] P 5e53385ddc904372a6685f865d7c9e28: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 21:26:12.956398 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: CompactRowSetsOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 2.038s user 1.249s sys 0.019s Metrics: {"bytes_written":614,"cfile_cache_hit":46,"cfile_cache_hit_bytes":889566,"cfile_cache_miss":60,"cfile_cache_miss_bytes":1314390,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":6357,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":4958,"drs_written":1,"lbm_read_time_us":3781,"lbm_reads_lt_1ms":92,"lbm_write_time_us":51068,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":78,"num_input_rowsets":2,"peak_mem_usage":1067657,"reinsert_count":0,"rows_written":32210,"thread_start_us":6500,"threads_started":1,"update_count":66}
I20240502 21:26:12.957377 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b): 605857 bytes on disk
I20240502 21:26:12.960778 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: UndoDeltaBlockGCOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 21:26:12.961683 15914 maintenance_manager.cc:419] P 1df45829447d4299992c186d49363fb4: Scheduling MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b): perf score=0.001197
I20240502 21:26:13.115341 15804 maintenance_manager.cc:643] P 1df45829447d4299992c186d49363fb4: MajorDeltaCompactionOp(26d7d451ab3b430dab914b22d4dbb83b) complete. Timing: real 0.153s user 0.131s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":876,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":247,"lbm_read_time_us":425,"lbm_reads_lt_1ms":20,"lbm_write_time_us":551,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":66}
I20240502 21:26:13.317379 15550 linked_list-test-util.h:767] Done collecting results (19000 rows in 881.556ms)
W20240502 21:26:13.330226 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:13.361449 15550 linked_list-test-util.h:556] Snapshot: P: 1714685152073191 usec, L: 1 verified. Result: OK
I20240502 21:26:13.366317 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685162088980 usec, L: 1 Expected Rows: 32500
I20240502 21:26:13.381467 15550 linked_list-test-util.h:528] Calling callback on tserver 1df45829447d4299992c186d49363fb4
I20240502 21:26:13.666131 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:13.960112 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:26:14.216737 15550 linked_list-test-util.h:767] Done collecting results (32500 rows in 835.43ms)
I20240502 21:26:14.294301 15550 linked_list-test-util.h:556] Snapshot: P: 1714685162088980 usec, L: 1 verified. Result: OK
I20240502 21:26:14.298025 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685172101048 usec, L: 1 Expected Rows: 51250
I20240502 21:26:14.308001 15550 linked_list-test-util.h:528] Calling callback on tserver 1df45829447d4299992c186d49363fb4
W20240502 21:26:14.351521 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:14.398759 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:14.683519 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:15.505501 15550 linked_list-test-util.h:767] Done collecting results (51250 rows in 1197.63ms)
I20240502 21:26:15.565186 15550 linked_list-test-util.h:556] Snapshot: P: 1714685172101048 usec, L: 1 verified. Result: OK
I20240502 21:26:15.568356 15550 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 51250
I20240502 21:26:15.680884 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:15.929999 15803 consensus_peers.cc:485] T 26d7d451ab3b430dab914b22d4dbb83b P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:26:16.411270 16096 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 21:26:16.892696 15803 consensus_peers.cc:485] T 23c3098e6fc94593a4adbd5b89076b2c P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:17.055938 15550 linked_list-test-util.h:767] Done collecting results (51250 rows in 1478.7ms)
I20240502 21:26:17.173154 15550 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240502 21:26:17.173799 15550 linked_list-test-util.h:692] Successfully verified 51250 rows
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs:
TabletServer: 7575b61d310d4850bbf16b4827e92922, Rpc address: host: "127.15.47.129" port: 38827 (Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111))
I20240502 21:26:17.197871 15550 webui_checker.cc:88] shutting down CURL thread
W20240502 21:26:17.355937 15803 consensus_peers.cc:485] T ce42e0503d2c4b08a3a825f2d1e6fd01 P 1df45829447d4299992c186d49363fb4 -> Peer 7575b61d310d4850bbf16b4827e92922 (127.15.47.129:38827): Couldn't send request to peer 7575b61d310d4850bbf16b4827e92922. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:38827: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:17.592053 16127 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 21:26:18.175084 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 15785
I20240502 21:26:18.176568 15907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.441143 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 15785
I20240502 21:26:18.503481 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 15916
I20240502 21:26:18.504554 16040 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.744359 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 15916
I20240502 21:26:18.799691 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 15562
I20240502 21:26:18.801060 15623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:18.930907 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 15562
2024-05-02T21:26:18Z chronyd exiting
I20240502 21:26:18.976608 15550 test_util.cc:175] -----------------------------------------------
I20240502 21:26:18.976737 15550 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714685139239457-15550-0
[ FAILED ] LinkedListTest.TestLoadAndVerify (39653 ms)
[ RUN ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240502 21:26:18.978741 15550 test_util.cc:268] Using random seed: 1090429344
I20240502 21:26:18.980511 15550 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 21:26:18.980578 15550 linked_list-test.cc:101] --------------
I20240502 21:26:18.980648 15550 linked_list-test.cc:102] 50 chains
I20240502 21:26:18.980692 15550 linked_list-test.cc:103] 3 tablets
I20240502 21:26:18.980765 15550 linked_list-test.cc:104] Mutations on
I20240502 21:26:18.980814 15550 linked_list-test.cc:105] --------------
I20240502 21:26:18.980980 15550 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:18.981029 15550 ts_itest-base.cc:113] --------------
I20240502 21:26:18.981097 15550 ts_itest-base.cc:114] 3 tablet servers
I20240502 21:26:18.981150 15550 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:26:18.981212 15550 ts_itest-base.cc:116] --------------
2024-05-02T21:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:18Z Disabled control of system clock
I20240502 21:26:19.017647 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.47.190:33697
--webserver_interface=127.15.47.190
--webserver_port=0
--builtin_ntp_servers=127.15.47.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.47.190:33697
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 21:26:19.384526 16380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:19.384860 16380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:19.385010 16380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:19.392498 16380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:19.392628 16380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:19.392722 16380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:19.392807 16380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:19.401939 16380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.47.190:33697
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.47.190:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.15.47.190
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:19.403404 16380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:19.405278 16380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:19.417541 16385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:19.418489 16388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:19.419324 16386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:19.420089 16380 server_base.cc:1034] running on GCE node
I20240502 21:26:19.420917 16380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:19.422363 16380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:19.423573 16380 hybrid_clock.cc:648] HybridClock initialized: now 1714685179423500 us; error 52 us; skew 500 ppm
I20240502 21:26:19.423897 16380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:19.426170 16380 webserver.cc:460] Webserver started at http://127.15.47.190:42575/ using document root <none> and password file <none>
I20240502 21:26:19.426910 16380 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:19.427037 16380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:19.427353 16380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:19.430136 16380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/instance:
uuid: "7dc3a43117714815a86ead4b758c466d"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:19.430876 16380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal/instance:
uuid: "7dc3a43117714815a86ead4b758c466d"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:19.436424 16380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 21:26:19.440047 16394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:19.441524 16380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:19.441731 16380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
uuid: "7dc3a43117714815a86ead4b758c466d"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:19.441936 16380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:19.476984 16380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:19.477875 16380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:19.478170 16380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:19.496364 16380 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.190:33697
I20240502 21:26:19.496383 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.190:33697 every 8 connection(s)
I20240502 21:26:19.497825 16380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/data/info.pb
I20240502 21:26:19.501720 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.502550 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 16380
I20240502 21:26:19.502805 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/master-0/wal/instance
I20240502 21:26:19.514658 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d: Bootstrap starting.
I20240502 21:26:19.518277 16446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.519299 16446 log.cc:824] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:19.522289 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d: No bootstrap required, opened a new log
I20240502 21:26:19.528533 16446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } }
I20240502 21:26:19.528928 16446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.529044 16446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc3a43117714815a86ead4b758c466d, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.529657 16446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } }
I20240502 21:26:19.529870 16446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:19.529990 16446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:19.530155 16446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.532469 16446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } }
I20240502 21:26:19.532987 16446 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc3a43117714815a86ead4b758c466d; no voters:
I20240502 21:26:19.533560 16446 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:19.533715 16451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:19.534372 16451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [term 1 LEADER]: Becoming Leader. State: Replica: 7dc3a43117714815a86ead4b758c466d, State: Running, Role: LEADER
I20240502 21:26:19.534950 16451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } }
I20240502 21:26:19.535704 16446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:19.538022 16452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc3a43117714815a86ead4b758c466d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } } }
I20240502 21:26:19.538190 16453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc3a43117714815a86ead4b758c466d. Latest consensus state: current_term: 1 leader_uuid: "7dc3a43117714815a86ead4b758c466d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc3a43117714815a86ead4b758c466d" member_type: VOTER last_known_addr { host: "127.15.47.190" port: 33697 } } }
I20240502 21:26:19.538388 16452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:19.538437 16453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:19.541152 16458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:19.546155 16458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:19.555814 16458 catalog_manager.cc:1347] Generated new cluster ID: 231755cba1a74868b31014cff23ba447
I20240502 21:26:19.555943 16458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:19.580333 16458 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:19.582243 16458 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:19.600440 16458 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7dc3a43117714815a86ead4b758c466d: Generated new TSK 0
I20240502 21:26:19.601693 16458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:19.607448 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.129:0
--local_ip_for_outbound_sockets=127.15.47.129
--webserver_interface=127.15.47.129
--webserver_port=0
--tserver_master_addrs=127.15.47.190:33697
--builtin_ntp_servers=127.15.47.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 21:26:19.955013 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:19.955307 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:19.955428 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:19.963430 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:19.963658 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.129
I20240502 21:26:19.973903 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.15.47.129
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.15.47.190:33697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.129
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:19.976104 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:19.977967 16470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:19.991341 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:19.992087 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:19.993094 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:19.993932 16470 server_base.cc:1034] running on GCE node
I20240502 21:26:19.994567 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:19.995538 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:19.996755 16470 hybrid_clock.cc:648] HybridClock initialized: now 1714685179996698 us; error 75 us; skew 500 ppm
I20240502 21:26:19.997071 16470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:19.999348 16470 webserver.cc:460] Webserver started at http://127.15.47.129:38101/ using document root <none> and password file <none>
I20240502 21:26:20.000102 16470 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.000231 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.000571 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.003392 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/instance:
uuid: "b8f18a3c4ad84591ac8f11696299d572"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:20.004169 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal/instance:
uuid: "b8f18a3c4ad84591ac8f11696299d572"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:20.009550 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:20.013095 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.014523 16470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:20.014688 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
uuid: "b8f18a3c4ad84591ac8f11696299d572"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:20.014864 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.047350 16470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.048265 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:20.048606 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.049615 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:20.051167 16470 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:20.051249 16470 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.051373 16470 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:20.051432 16470 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.084250 16470 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.129:33261
I20240502 21:26:20.084324 16597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.129:33261 every 8 connection(s)
I20240502 21:26:20.085695 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
I20240502 21:26:20.093744 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 16470
I20240502 21:26:20.093984 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal/instance
I20240502 21:26:20.097599 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.130:0
--local_ip_for_outbound_sockets=127.15.47.130
--webserver_interface=127.15.47.130
--webserver_port=0
--tserver_master_addrs=127.15.47.190:33697
--builtin_ntp_servers=127.15.47.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 21:26:20.099443 16598 heartbeater.cc:347] Connected to a master server at 127.15.47.190:33697
I20240502 21:26:20.099763 16598 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:20.100760 16598 heartbeater.cc:511] Master 127.15.47.190:33697 requested a full tablet report, sending...
I20240502 21:26:20.102942 16411 ts_manager.cc:194] Registered new tserver with Master: b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261)
I20240502 21:26:20.104575 16411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.129:44057
W20240502 21:26:20.453011 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:20.453387 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:20.453516 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:20.461280 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:20.461499 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.130
I20240502 21:26:20.470819 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.15.47.130
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.15.47.190:33697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.130
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:20.472232 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:20.473989 16602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.486891 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.487813 16610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.488509 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.489252 16602 server_base.cc:1034] running on GCE node
I20240502 21:26:20.489882 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:20.490823 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:20.492048 16602 hybrid_clock.cc:648] HybridClock initialized: now 1714685180491982 us; error 38 us; skew 500 ppm
I20240502 21:26:20.492381 16602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.494910 16602 webserver.cc:460] Webserver started at http://127.15.47.130:44771/ using document root <none> and password file <none>
I20240502 21:26:20.495760 16602 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.495918 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.496325 16602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.499758 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/instance:
uuid: "9d166f3cfc3a429c98a9c93712bcd1f7"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:20.500687 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal/instance:
uuid: "9d166f3cfc3a429c98a9c93712bcd1f7"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:20.507268 16602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 21:26:20.511493 16616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.513186 16602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:20.513381 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
uuid: "9d166f3cfc3a429c98a9c93712bcd1f7"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:20.513576 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.545186 16602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.546066 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:20.546391 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.547449 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:20.549048 16602 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:20.549121 16602 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.549216 16602 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:20.549259 16602 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.580956 16602 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.130:35495
I20240502 21:26:20.581007 16729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.130:35495 every 8 connection(s)
I20240502 21:26:20.582278 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/data/info.pb
I20240502 21:26:20.591724 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 16602
I20240502 21:26:20.591922 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-1/wal/instance
I20240502 21:26:20.595294 16730 heartbeater.cc:347] Connected to a master server at 127.15.47.190:33697
I20240502 21:26:20.595496 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.131:0
--local_ip_for_outbound_sockets=127.15.47.131
--webserver_interface=127.15.47.131
--webserver_port=0
--tserver_master_addrs=127.15.47.190:33697
--builtin_ntp_servers=127.15.47.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 21:26:20.595649 16730 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:20.596547 16730 heartbeater.cc:511] Master 127.15.47.190:33697 requested a full tablet report, sending...
I20240502 21:26:20.598121 16411 ts_manager.cc:194] Registered new tserver with Master: 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495)
I20240502 21:26:20.599265 16411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.130:43243
W20240502 21:26:20.943367 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:20.943715 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:20.943884 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:20.951835 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:20.952087 16734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.131
I20240502 21:26:20.961571 16734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.15.47.131
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.15.47.190:33697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.131
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:20.963044 16734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:20.965018 16734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.979075 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.979580 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.980226 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.981305 16734 server_base.cc:1034] running on GCE node
I20240502 21:26:20.981976 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:20.983093 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:20.984380 16734 hybrid_clock.cc:648] HybridClock initialized: now 1714685180984264 us; error 75 us; skew 500 ppm
I20240502 21:26:20.984714 16734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.987017 16734 webserver.cc:460] Webserver started at http://127.15.47.131:40651/ using document root <none> and password file <none>
I20240502 21:26:20.987902 16734 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.988039 16734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.988389 16734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.991286 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/instance:
uuid: "6aeb1ddcee054db7b9aac7bb97c84d13"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:20.992074 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal/instance:
uuid: "6aeb1ddcee054db7b9aac7bb97c84d13"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:20.997567 16734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 21:26:21.001168 16748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.002521 16734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:21.002705 16734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
uuid: "6aeb1ddcee054db7b9aac7bb97c84d13"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-ntzt"
I20240502 21:26:21.002889 16734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:21.022073 16734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:21.022950 16734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:21.023247 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:21.024262 16734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:21.025889 16734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:21.025964 16734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.026094 16734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:21.026146 16734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:21.058904 16734 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.131:41785
I20240502 21:26:21.058935 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.131:41785 every 8 connection(s)
I20240502 21:26:21.060348 16734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/data/info.pb
I20240502 21:26:21.070705 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 16734
I20240502 21:26:21.070953 15550 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-2/wal/instance
I20240502 21:26:21.073627 16862 heartbeater.cc:347] Connected to a master server at 127.15.47.190:33697
I20240502 21:26:21.073956 16862 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:21.074826 16862 heartbeater.cc:511] Master 127.15.47.190:33697 requested a full tablet report, sending...
I20240502 21:26:21.076326 16411 ts_manager.cc:194] Registered new tserver with Master: 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
I20240502 21:26:21.077158 16411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.131:48897
I20240502 21:26:21.086872 15550 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:26:21.103266 16411 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
I20240502 21:26:21.108520 16598 heartbeater.cc:503] Master 127.15.47.190:33697 was elected leader, sending a full tablet report...
W20240502 21:26:21.109516 16411 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:26:21.138393 16665 tablet_service.cc:1467] Processing CreateTablet for tablet 6633081a181144f6924eb5e1eaf822c5 (DEFAULT_TABLE table=TestTable [id=346645e4de834ca0bfb685237b81edfe]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 21:26:21.138455 16532 tablet_service.cc:1467] Processing CreateTablet for tablet 6633081a181144f6924eb5e1eaf822c5 (DEFAULT_TABLE table=TestTable [id=346645e4de834ca0bfb685237b81edfe]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 21:26:21.139844 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6633081a181144f6924eb5e1eaf822c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.141032 16532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6633081a181144f6924eb5e1eaf822c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.140878 16797 tablet_service.cc:1467] Processing CreateTablet for tablet 6633081a181144f6924eb5e1eaf822c5 (DEFAULT_TABLE table=TestTable [id=346645e4de834ca0bfb685237b81edfe]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 21:26:21.142366 16797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6633081a181144f6924eb5e1eaf822c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.151091 16880 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Bootstrap starting.
I20240502 21:26:21.155216 16880 tablet_bootstrap.cc:654] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:21.156355 16882 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Bootstrap starting.
I20240502 21:26:21.156713 16880 log.cc:824] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:21.157382 16881 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Bootstrap starting.
I20240502 21:26:21.161482 16882 tablet_bootstrap.cc:654] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:21.161482 16881 tablet_bootstrap.cc:654] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:21.162210 16880 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: No bootstrap required, opened a new log
I20240502 21:26:21.162590 16880 ts_tablet_manager.cc:1367] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 21:26:21.163019 16882 log.cc:824] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:21.163933 16881 log.cc:824] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:21.166368 16882 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: No bootstrap required, opened a new log
I20240502 21:26:21.166692 16882 ts_tablet_manager.cc:1367] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 21:26:21.166872 16881 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: No bootstrap required, opened a new log
I20240502 21:26:21.167173 16881 ts_tablet_manager.cc:1367] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240502 21:26:21.169764 16880 raft_consensus.cc:348] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.170166 16880 raft_consensus.cc:374] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:21.170286 16880 raft_consensus.cc:729] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d166f3cfc3a429c98a9c93712bcd1f7, State: Initialized, Role: FOLLOWER
I20240502 21:26:21.170929 16880 consensus_queue.cc:260] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.172114 16730 heartbeater.cc:503] Master 127.15.47.190:33697 was elected leader, sending a full tablet report...
I20240502 21:26:21.173452 16880 ts_tablet_manager.cc:1398] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240502 21:26:21.176162 16882 raft_consensus.cc:348] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.176746 16882 raft_consensus.cc:374] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:21.176890 16882 raft_consensus.cc:729] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aeb1ddcee054db7b9aac7bb97c84d13, State: Initialized, Role: FOLLOWER
I20240502 21:26:21.177253 16881 raft_consensus.cc:348] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.177532 16882 consensus_queue.cc:260] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.177723 16881 raft_consensus.cc:374] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:21.177848 16881 raft_consensus.cc:729] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f18a3c4ad84591ac8f11696299d572, State: Initialized, Role: FOLLOWER
I20240502 21:26:21.178642 16881 consensus_queue.cc:260] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.178802 16862 heartbeater.cc:503] Master 127.15.47.190:33697 was elected leader, sending a full tablet report...
I20240502 21:26:21.180557 16881 ts_tablet_manager.cc:1398] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20240502 21:26:21.181615 16882 ts_tablet_manager.cc:1398] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240502 21:26:21.243047 16888 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:21.243320 16888 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.245085 16888 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495), 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
I20240502 21:26:21.252712 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "b8f18a3c4ad84591ac8f11696299d572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" is_pre_election: true
I20240502 21:26:21.253034 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "b8f18a3c4ad84591ac8f11696299d572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" is_pre_election: true
I20240502 21:26:21.253252 16685 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f18a3c4ad84591ac8f11696299d572 in term 0.
I20240502 21:26:21.253475 16817 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f18a3c4ad84591ac8f11696299d572 in term 0.
I20240502 21:26:21.253957 16487 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d166f3cfc3a429c98a9c93712bcd1f7, b8f18a3c4ad84591ac8f11696299d572; no voters:
I20240502 21:26:21.254335 16888 raft_consensus.cc:2781] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:21.254478 16888 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:21.254590 16888 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:21.256845 16888 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.257530 16888 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 1 election: Requested vote from peers 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495), 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
I20240502 21:26:21.258029 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "b8f18a3c4ad84591ac8f11696299d572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7"
I20240502 21:26:21.258107 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "b8f18a3c4ad84591ac8f11696299d572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13"
I20240502 21:26:21.258231 16685 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:21.258284 16817 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:21.260411 16817 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f18a3c4ad84591ac8f11696299d572 in term 1.
I20240502 21:26:21.260422 16685 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f18a3c4ad84591ac8f11696299d572 in term 1.
I20240502 21:26:21.260823 16487 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d166f3cfc3a429c98a9c93712bcd1f7, b8f18a3c4ad84591ac8f11696299d572; no voters:
I20240502 21:26:21.261132 16888 raft_consensus.cc:2781] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:21.261562 16888 raft_consensus.cc:686] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 LEADER]: Becoming Leader. State: Replica: b8f18a3c4ad84591ac8f11696299d572, State: Running, Role: LEADER
I20240502 21:26:21.262074 16888 consensus_queue.cc:237] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:21.267114 16411 catalog_manager.cc:5484] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129). New cstate: current_term: 1 leader_uuid: "b8f18a3c4ad84591ac8f11696299d572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:21.280686 15550 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:26:21.282864 15550 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b8f18a3c4ad84591ac8f11696299d572 to finish bootstrapping
I20240502 21:26:21.291682 15550 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9d166f3cfc3a429c98a9c93712bcd1f7 to finish bootstrapping
I20240502 21:26:21.299203 15550 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6aeb1ddcee054db7b9aac7bb97c84d13 to finish bootstrapping
I20240502 21:26:21.306293 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 16470
I20240502 21:26:21.307353 16593 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:21.312150 16863 tablet.cc:2369] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:21.334424 16731 tablet.cc:2369] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:21.395925 16599 tablet.cc:2369] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:21.396754 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 16470
I20240502 21:26:21.420523 15550 linked_list-test-util.h:401] Running for: 30.000s
I20240502 21:26:21.420656 15550 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 21:26:21.420758 15550 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
W20240502 21:26:21.427294 16867 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261) has failed: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111)
W20240502 21:26:21.441009 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.457300 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.473706 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.482901 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.508378 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.520826 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.546149 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.563714 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.596928 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.616416 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.658900 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.678148 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.720932 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.747155 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.800555 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.825683 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.886893 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.916163 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:21.983371 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.019021 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.086359 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.123392 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.197346 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.237602 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.324491 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.368847 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.457376 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.501857 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.548012 16867 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261) has failed: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111)
W20240502 21:26:22.593973 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.643343 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.744633 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.797128 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.903777 16645 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59976: Illegal state: replica 9d166f3cfc3a429c98a9c93712bcd1f7 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.960216 16777 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45014: Illegal state: replica 6aeb1ddcee054db7b9aac7bb97c84d13 is not leader of this config: current role FOLLOWER
I20240502 21:26:22.979522 16897 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:22.979744 16897 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:22.981384 16897 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261), 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495)
W20240502 21:26:22.983197 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111)
W20240502 21:26:22.984938 16751 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111)
I20240502 21:26:22.988605 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" is_pre_election: true
I20240502 21:26:22.988883 16685 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aeb1ddcee054db7b9aac7bb97c84d13 in term 1.
I20240502 21:26:22.989449 16751 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6aeb1ddcee054db7b9aac7bb97c84d13, 9d166f3cfc3a429c98a9c93712bcd1f7; no voters: b8f18a3c4ad84591ac8f11696299d572
I20240502 21:26:22.989868 16897 raft_consensus.cc:2781] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:26:22.990026 16897 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:22.990132 16897 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.992383 16897 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:22.993014 16897 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 election: Requested vote from peers b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261), 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495)
I20240502 21:26:22.994004 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6633081a181144f6924eb5e1eaf822c5" candidate_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7"
I20240502 21:26:22.994315 16685 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:22.994562 16751 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111)
I20240502 21:26:22.996484 16685 raft_consensus.cc:2454] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aeb1ddcee054db7b9aac7bb97c84d13 in term 2.
I20240502 21:26:22.996922 16751 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6aeb1ddcee054db7b9aac7bb97c84d13, 9d166f3cfc3a429c98a9c93712bcd1f7; no voters: b8f18a3c4ad84591ac8f11696299d572
I20240502 21:26:22.997267 16897 raft_consensus.cc:2781] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:22.997692 16897 raft_consensus.cc:686] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [term 2 LEADER]: Becoming Leader. State: Replica: 6aeb1ddcee054db7b9aac7bb97c84d13, State: Running, Role: LEADER
I20240502 21:26:22.998286 16897 consensus_queue.cc:237] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:23.003042 16408 catalog_manager.cc:5484] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 reported cstate change: term changed from 1 to 2, leader changed from b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129) to 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131). New cstate: current_term: 2 leader_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:23.021322 16685 raft_consensus.cc:1261] T 6633081a181144f6924eb5e1eaf822c5 P 9d166f3cfc3a429c98a9c93712bcd1f7 [term 2 FOLLOWER]: Refusing update from remote peer 6aeb1ddcee054db7b9aac7bb97c84d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 21:26:23.021595 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:23.022321 16902 consensus_queue.cc:1035] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:23.030194 16909 mvcc.cc:205] Tried to move back new op lower bound from 7023350509643583488 to 7023350509565067264. Current Snapshot: MvccSnapshot[applied={T|T < 7023350509643583488}]
I20240502 21:26:23.030853 16911 mvcc.cc:205] Tried to move back new op lower bound from 7023350509643583488 to 7023350509565067264. Current Snapshot: MvccSnapshot[applied={T|T < 7023350509643583488}]
W20240502 21:26:25.292562 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:26:27.636476 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:26:28.037726 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 21:26:28.415552 16902 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 21 ops behind the committed index
W20240502 21:26:29.937330 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:26:31.425293 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350544057413632 Inserted count: 40700
I20240502 21:26:31.585729 16915 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 333 ops behind the committed index [suppressed 5 similar messages]
W20240502 21:26:32.456444 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:26:33.522684 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:26:35.149812 16902 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 691 ops behind the committed index [suppressed 6 similar messages]
W20240502 21:26:35.157459 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:26:37.933781 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:26:38.372707 16915 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 1006 ops behind the committed index [suppressed 5 similar messages]
W20240502 21:26:38.886289 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:26:40.270366 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:41.467413 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350585157611520 Inserted count: 89900
I20240502 21:26:41.788705 16897 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 1340 ops behind the committed index [suppressed 6 similar messages]
W20240502 21:26:42.866261 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:26:44.342758 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:26:44.904408 16915 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 1643 ops behind the committed index [suppressed 5 similar messages]
W20240502 21:26:45.373227 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:26:47.933246 17020 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 1944 ops behind the committed index [suppressed 5 similar messages]
W20240502 21:26:47.941234 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:26:49.633550 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:26:50.686471 16751 consensus_peers.cc:485] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 -> Peer b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261): Couldn't send request to peer b8f18a3c4ad84591ac8f11696299d572. Status: Network error: Client connection negotiation failed: client connection to 127.15.47.129:33261: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:26:51.069741 17065 consensus_queue.cc:786] T 6633081a181144f6924eb5e1eaf822c5 P 6aeb1ddcee054db7b9aac7bb97c84d13 [LEADER]: Peer b8f18a3c4ad84591ac8f11696299d572 is lagging by at least 2263 ops behind the committed index [suppressed 5 similar messages]
I20240502 21:26:51.425504 15550 linked_list-test-util.h:417] Sample at HT timestamp: 7023350625980284928 Inserted count: 139100
I20240502 21:26:51.425679 15550 linked_list-test-util.h:421] Finished inserting list. Added 139100 in chain
I20240502 21:26:51.425791 15550 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 21:26:51.425904 15550 linked_list-test-util.h:424] 0: 7153446030691401728
I20240502 21:26:51.425966 15550 linked_list-test-util.h:424] 1: 445587945318776833
I20240502 21:26:51.426033 15550 linked_list-test-util.h:424] 2: -6262129402565558270
I20240502 21:26:51.426101 15550 linked_list-test-util.h:424] 3: 5476756585771368451
I20240502 21:26:51.426165 15550 linked_list-test-util.h:424] 4: 5922626006066855940
I20240502 21:26:51.426216 15550 linked_list-test-util.h:424] 5: -785232079305768955
I20240502 21:26:51.426278 15550 linked_list-test-util.h:424] 6: -7493090164678393850
I20240502 21:26:51.426329 15550 linked_list-test-util.h:424] 7: 4245795823658532871
I20240502 21:26:51.426399 15550 linked_list-test-util.h:424] 8: 4691383768977309704
I20240502 21:26:51.426455 15550 linked_list-test-util.h:424] 9: -2016474316395315191
I20240502 21:26:51.426517 15550 linked_list-test-util.h:424] 10: -8724191664279650294
I20240502 21:26:51.426566 15550 linked_list-test-util.h:424] 11: 3014975799033987083
I20240502 21:26:51.426653 15550 linked_list-test-util.h:424] 12: 3460563744352763916
I20240502 21:26:51.426716 15550 linked_list-test-util.h:424] 13: -3247294341019860979
I20240502 21:26:51.426784 15550 linked_list-test-util.h:424] 14: 8491591647317065742
I20240502 21:26:51.426857 15550 linked_list-test-util.h:424] 15: 1783733561944440847
I20240502 21:26:51.426934 15550 linked_list-test-util.h:424] 16: 2229602982239928336
I20240502 21:26:51.426990 15550 linked_list-test-util.h:424] 17: -4478255103132696559
I20240502 21:26:51.427058 15550 linked_list-test-util.h:424] 18: 7260771622692519954
I20240502 21:26:51.427114 15550 linked_list-test-util.h:424] 19: 552913537319895059
I20240502 21:26:51.427181 15550 linked_list-test-util.h:424] 20: 998501482638671892
I20240502 21:26:51.427237 15550 linked_list-test-util.h:424] 21: -5709075127757242347
I20240502 21:26:51.427305 15550 linked_list-test-util.h:424] 22: 6029810860579684374
I20240502 21:26:51.427379 15550 linked_list-test-util.h:424] 23: -678047224792940521
I20240502 21:26:51.427451 15550 linked_list-test-util.h:424] 24: -232459279474163688
I20240502 21:26:51.427508 15550 linked_list-test-util.h:424] 25: -6940317364846788583
I20240502 21:26:51.427578 15550 linked_list-test-util.h:424] 26: 4798709360978427930
I20240502 21:26:51.427632 15550 linked_list-test-util.h:424] 27: 5244578781273915419
I20240502 21:26:51.427709 15550 linked_list-test-util.h:424] 28: -1463279304098709476
I20240502 21:26:51.427778 15550 linked_list-test-util.h:424] 29: -8171137389471334371
I20240502 21:26:51.427846 15550 linked_list-test-util.h:424] 30: 3567748598865592350
I20240502 21:26:51.427906 15550 linked_list-test-util.h:424] 31: 4013336544184369183
I20240502 21:26:51.427976 15550 linked_list-test-util.h:424] 32: -2694240066211545056
I20240502 21:26:51.428030 15550 linked_list-test-util.h:424] 33: 9044645922125381665
I20240502 21:26:51.428095 15550 linked_list-test-util.h:424] 34: 2336928574241046562
I20240502 21:26:51.428148 15550 linked_list-test-util.h:424] 35: 2782516519559823395
I20240502 21:26:51.428216 15550 linked_list-test-util.h:424] 36: -3925341565812801500
I20240502 21:26:51.428296 15550 linked_list-test-util.h:424] 37: 7813544422524125221
I20240502 21:26:51.428371 15550 linked_list-test-util.h:424] 38: 1105686337151500326
I20240502 21:26:51.428426 15550 linked_list-test-util.h:424] 39: 1551555757446987815
I20240502 21:26:51.428504 15550 linked_list-test-util.h:424] 40: -5156302327925637080
I20240502 21:26:51.428556 15550 linked_list-test-util.h:424] 41: 6582583660411289641
I20240502 21:26:51.428619 15550 linked_list-test-util.h:424] 42: -125133687473045462
I20240502 21:26:51.428674 15550 linked_list-test-util.h:424] 43: 320735732822442027
I20240502 21:26:51.428766 15550 linked_list-test-util.h:424] 44: -6387122352550182868
I20240502 21:26:51.428819 15550 linked_list-test-util.h:424] 45: 5351763635786743853
I20240502 21:26:51.428893 15550 linked_list-test-util.h:424] 46: -1356094449585881042
I20240502 21:26:51.428962 15550 linked_list-test-util.h:424] 47: -910506504267104209
I20240502 21:26:51.429049 15550 linked_list-test-util.h:424] 48: -7618364589639729104
I20240502 21:26:51.429105 15550 linked_list-test-util.h:424] 49: 4120521398697197617
I20240502 21:26:51.535807 15550 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
/tmp/dist-test-taskgN4F9I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.47.129:33261
--local_ip_for_outbound_sockets=127.15.47.129
--tserver_master_addrs=127.15.47.190:33697
--webserver_port=38101
--webserver_interface=127.15.47.129
--builtin_ntp_servers=127.15.47.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 21:26:51.905187 17077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:51.905505 17077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:51.905687 17077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:51.913306 17077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:51.913551 17077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.47.129
I20240502 21:26:51.922461 17077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.47.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.47.129:33261
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.15.47.129
--webserver_port=38101
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.15.47.190:33697
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.47.129
--log_dir=/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:51.923851 17077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:51.925691 17077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:51.938158 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.938340 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.939365 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:51.939693 17077 server_base.cc:1034] running on GCE node
I20240502 21:26:51.940353 17077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:51.941380 17077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:51.942605 17077 hybrid_clock.cc:648] HybridClock initialized: now 1714685211942534 us; error 48 us; skew 500 ppm
I20240502 21:26:51.942927 17077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:51.952327 17077 webserver.cc:460] Webserver started at http://127.15.47.129:38101/ using document root <none> and password file <none>
I20240502 21:26:51.953151 17077 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:51.953308 17077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:51.959857 17077 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:26:51.963798 17091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.965469 17077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:26:51.965641 17077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
uuid: "b8f18a3c4ad84591ac8f11696299d572"
format_stamp: "Formatted at 2024-05-02 21:26:19 on dist-test-slave-ntzt"
I20240502 21:26:51.966521 17077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:51.993618 17077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:51.994520 17077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:51.994813 17077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:51.995822 17077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:51.997815 17098 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 21:26:52.003407 17077 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 21:26:52.003518 17077 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20240502 21:26:52.003660 17077 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 21:26:52.006588 17077 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 21:26:52.006668 17077 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:52.007071 17098 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Bootstrap starting.
I20240502 21:26:52.030195 17098 log.cc:824] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:52.034613 17098 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:52.035072 17098 log.cc:1225] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/wal/wals/6633081a181144f6924eb5e1eaf822c5.recovery-1714685212034956
I20240502 21:26:52.035205 17098 tablet_bootstrap.cc:492] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Bootstrap complete.
I20240502 21:26:52.035854 17098 ts_tablet_manager.cc:1367] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.015s
I20240502 21:26:52.049095 17098 raft_consensus.cc:348] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:52.050904 17098 raft_consensus.cc:729] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f18a3c4ad84591ac8f11696299d572, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.052112 17098 consensus_queue.cc:260] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:52.052521 17077 rpc_server.cc:307] RPC server started. Bound to: 127.15.47.129:33261
I20240502 21:26:52.052559 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.47.129:33261 every 8 connection(s)
I20240502 21:26:52.053982 17077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgN4F9I/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714685139239457-15550-0/linked-list-cluster/ts-0/data/info.pb
I20240502 21:26:52.056604 17098 ts_tablet_manager.cc:1398] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20240502 21:26:52.063891 15550 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu as pid 17077
I20240502 21:26:52.068038 17208 heartbeater.cc:347] Connected to a master server at 127.15.47.190:33697
I20240502 21:26:52.068645 17208 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:52.080413 17208 heartbeater.cc:511] Master 127.15.47.190:33697 requested a full tablet report, sending...
I20240502 21:26:52.087663 16408 ts_manager.cc:194] Re-registered known tserver with Master: b8f18a3c4ad84591ac8f11696299d572 (127.15.47.129:33261)
I20240502 21:26:52.089491 16408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.47.129:49573
I20240502 21:26:52.092967 17208 heartbeater.cc:503] Master 127.15.47.190:33697 was elected leader, sending a full tablet report...
I20240502 21:26:52.112334 15550 cluster_itest_util.cc:256] Not converged past 2782 yet: 1.1 2.2811 2.2811
I20240502 21:26:52.130725 17160 raft_consensus.cc:3037] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:52.135412 17160 pending_rounds.cc:76] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Aborting all ops after (but not including) 0
I20240502 21:26:52.135572 17160 pending_rounds.cc:97] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 21:26:52.135727 17160 raft_consensus.cc:2866] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:26:52.258574 15550 cluster_itest_util.cc:256] Not converged past 2782 yet: 2.506 2.2811 2.2811
I20240502 21:26:52.539386 15550 cluster_itest_util.cc:256] Not converged past 2782 yet: 2.1011 2.2811 2.2811
I20240502 21:26:52.843017 15550 cluster_itest_util.cc:256] Not converged past 2782 yet: 2.1513 2.2811 2.2811
I20240502 21:26:53.088238 17217 mvcc.cc:205] Tried to move back new op lower bound from 7023350530782081024 to 7023350509565067264. Current Snapshot: MvccSnapshot[applied={T|T < 7023350530782081024}]
I20240502 21:26:53.392892 15550 cluster_itest_util.cc:256] Not converged past 2782 yet: 2.2521 2.2811 2.2811
I20240502 21:26:53.896950 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 16602
I20240502 21:26:53.898432 16725 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:54.237566 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 16602
W20240502 21:26:54.300135 16395 connection.cc:535] server connection from 127.15.47.130:43243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:54.300628 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 16734
I20240502 21:26:54.301892 16857 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:54.667064 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 16734
W20240502 21:26:54.735034 17095 connection.cc:535] server connection from 127.15.47.131:54469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:26:54.736472 16397 connection.cc:535] server connection from 127.15.47.131:48897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:54.741015 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685191420267 usec, L: 1 Expected Rows: 40700
I20240502 21:26:54.765427 15550 meta_cache.cc:1510] marking tablet server 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) as failed
W20240502 21:26:54.765656 15550 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:26:54.801074 15550 linked_list-test-util.h:528] Calling callback on tserver b8f18a3c4ad84591ac8f11696299d572
I20240502 21:26:56.168295 17233 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6aeb1ddcee054db7b9aac7bb97c84d13)
I20240502 21:26:56.168699 17233 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:56.170822 17233 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495), 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
W20240502 21:26:56.173563 17094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.47.130:35495: connect: Connection refused (error 111)
W20240502 21:26:56.179615 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495): Network error: Client connection negotiation failed: client connection to 127.15.47.130:35495: connect: Connection refused (error 111)
W20240502 21:26:56.180769 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785): Network error: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:26:56.180990 17094 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f18a3c4ad84591ac8f11696299d572; no voters: 6aeb1ddcee054db7b9aac7bb97c84d13, 9d166f3cfc3a429c98a9c93712bcd1f7
I20240502 21:26:56.181448 17233 raft_consensus.cc:2726] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:26:56.339845 15550 linked_list-test-util.h:767] Done collecting results (40700 rows in 1538.96ms)
I20240502 21:26:56.406095 15550 linked_list-test-util.h:556] Snapshot: P: 1714685191420267 usec, L: 1 verified. Result: OK
I20240502 21:26:56.409618 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685201454495 usec, L: 1 Expected Rows: 89900
I20240502 21:26:56.411949 15550 meta_cache.cc:1510] marking tablet server 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) as failed
W20240502 21:26:56.412083 15550 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:26:56.432734 15550 linked_list-test-util.h:528] Calling callback on tserver b8f18a3c4ad84591ac8f11696299d572
I20240502 21:26:57.730034 17237 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6aeb1ddcee054db7b9aac7bb97c84d13)
I20240502 21:26:57.730242 17237 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:26:57.730877 17237 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495), 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
W20240502 21:26:57.733103 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495): Network error: Client connection negotiation failed: client connection to 127.15.47.130:35495: connect: Connection refused (error 111)
W20240502 21:26:57.733327 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785): Network error: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:26:57.733453 17094 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f18a3c4ad84591ac8f11696299d572; no voters: 6aeb1ddcee054db7b9aac7bb97c84d13, 9d166f3cfc3a429c98a9c93712bcd1f7
I20240502 21:26:57.733831 17237 raft_consensus.cc:2726] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:26:58.526826 15550 linked_list-test-util.h:767] Done collecting results (89900 rows in 2094.25ms)
I20240502 21:26:58.634425 15550 linked_list-test-util.h:556] Snapshot: P: 1714685201454495 usec, L: 1 verified. Result: OK
I20240502 21:26:58.637738 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 1714685211420968 usec, L: 1 Expected Rows: 139100
I20240502 21:26:58.639380 15550 meta_cache.cc:1510] marking tablet server 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) as failed
W20240502 21:26:58.639483 15550 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:26:58.648699 15550 linked_list-test-util.h:528] Calling callback on tserver b8f18a3c4ad84591ac8f11696299d572
I20240502 21:27:00.303117 17242 raft_consensus.cc:482] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6aeb1ddcee054db7b9aac7bb97c84d13)
I20240502 21:27:00.303280 17242 raft_consensus.cc:504] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f18a3c4ad84591ac8f11696299d572" member_type: VOTER last_known_addr { host: "127.15.47.129" port: 33261 } } peers { permanent_uuid: "9d166f3cfc3a429c98a9c93712bcd1f7" member_type: VOTER last_known_addr { host: "127.15.47.130" port: 35495 } } peers { permanent_uuid: "6aeb1ddcee054db7b9aac7bb97c84d13" member_type: VOTER last_known_addr { host: "127.15.47.131" port: 41785 } }
I20240502 21:27:00.303977 17242 leader_election.cc:288] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495), 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785)
W20240502 21:27:00.311908 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9d166f3cfc3a429c98a9c93712bcd1f7 (127.15.47.130:35495): Network error: Client connection negotiation failed: client connection to 127.15.47.130:35495: connect: Connection refused (error 111)
W20240502 21:27:00.312130 17094 leader_election.cc:334] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785): Network error: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:27:00.312211 17094 leader_election.cc:302] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f18a3c4ad84591ac8f11696299d572; no voters: 6aeb1ddcee054db7b9aac7bb97c84d13, 9d166f3cfc3a429c98a9c93712bcd1f7
I20240502 21:27:00.312522 17242 raft_consensus.cc:2726] T 6633081a181144f6924eb5e1eaf822c5 P b8f18a3c4ad84591ac8f11696299d572 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:00.625674 15550 linked_list-test-util.h:767] Done collecting results (139100 rows in 1977.09ms)
I20240502 21:27:00.793236 15550 linked_list-test-util.h:556] Snapshot: P: 1714685211420968 usec, L: 1 verified. Result: OK
I20240502 21:27:00.796491 15550 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 139100
I20240502 21:27:00.798053 15550 meta_cache.cc:1510] marking tablet server 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) as failed
W20240502 21:27:00.798167 15550 meta_cache.cc:302] tablet 6633081a181144f6924eb5e1eaf822c5: replica 6aeb1ddcee054db7b9aac7bb97c84d13 (127.15.47.131:41785) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.15.47.131:41785: connect: Connection refused (error 111)
I20240502 21:27:00.802309 15550 linked_list-test-util.h:528] Calling callback on tserver b8f18a3c4ad84591ac8f11696299d572
I20240502 21:27:02.736434 15550 linked_list-test-util.h:767] Done collecting results (139100 rows in 1934.25ms)
I20240502 21:27:02.942214 15550 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240502 21:27:02.943084 15550 linked_list-test-util.h:692] Successfully verified 139100 rows
I20240502 21:27:02.944218 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 17077
I20240502 21:27:02.951169 17201 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:03.188508 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 17077
I20240502 21:27:03.241204 15550 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu pid 16380
I20240502 21:27:03.242398 16441 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:03.357306 15550 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgN4F9I/build/asan/bin/kudu with pid 16380
2024-05-02T21:27:03Z chronyd exiting
[ OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (44428 ms)
[----------] 2 tests from LinkedListTest (84082 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (84082 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] LinkedListTest.TestLoadAndVerify
1 FAILED TEST