LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 133f2399e9b8473fbff0dfcab9a3d022, Rpc address: host: "127.29.83.66" port: 39503 (Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111))
I20240502 07:58:44.187458 30029 webui_checker.cc:88] shutting down CURL thread
I20240502 07:58:44.506026 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:44.525735 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:45.271617 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:45.328649 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:45.407672 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30132
I20240502 07:58:45.419142 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30394
I20240502 07:58:45.427450 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30041
2024-05-02T07:58:45Z chronyd exiting
I20240502 07:58:45.445573 30029 test_util.cc:175] -----------------------------------------------
I20240502 07:58:45.445643 30029 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 07:58:08.576848 30029 test_util.cc:268] Using random seed: -155332294
I20240502 07:58:08.583535 30029 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 07:58:08.583596 30029 linked_list-test.cc:101] --------------
I20240502 07:58:08.583616 30029 linked_list-test.cc:102] 50 chains
I20240502 07:58:08.583632 30029 linked_list-test.cc:103] 3 tablets
I20240502 07:58:08.583654 30029 linked_list-test.cc:104] Mutations on
I20240502 07:58:08.583680 30029 linked_list-test.cc:105] --------------
I20240502 07:58:08.583765 30029 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:08.583791 30029 ts_itest-base.cc:113] --------------
I20240502 07:58:08.583807 30029 ts_itest-base.cc:114] 3 tablet servers
I20240502 07:58:08.583828 30029 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:08.583901 30029 ts_itest-base.cc:116] --------------
2024-05-02T07:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:08Z Disabled control of system clock
I20240502 07:58:08.593708 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.126:45947
--webserver_interface=127.29.83.126
--webserver_port=0
--builtin_ntp_servers=127.29.83.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.126:45947
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 07:58:08.700300 30041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:08.700556 30041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:08.700635 30041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:08.703665 30041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:08.703727 30041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:08.703758 30041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:08.703786 30041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:08.707451 30041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.126:45947
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.126:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.29.83.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.30041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:08.708392 30041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.709534 30041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:58:08.718185 30041 server_base.cc:1034] running on GCE node
W20240502 07:58:08.718504 30049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.719071 30046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.719302 30047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.719846 30041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:08.720795 30041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:08.721966 30041 hybrid_clock.cc:648] HybridClock initialized: now 1714636688721946 us; error 33 us; skew 500 ppm
I20240502 07:58:08.723883 30041 webserver.cc:460] Webserver started at http://127.29.83.126:38175/ using document root <none> and password file <none>
I20240502 07:58:08.724450 30041 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.724509 30041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.724680 30041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.726322 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/instance:
uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.726794 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal/instance:
uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.730264 30041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20240502 07:58:08.732620 30055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:08.733621 30041 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240502 07:58:08.733737 30041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.733824 30041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.758917 30041 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:08.759613 30041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:08.759773 30041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.766515 30041 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.126:45947
I20240502 07:58:08.766535 30107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.126:45947 every 8 connection(s)
I20240502 07:58:08.767478 30041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
I20240502 07:58:08.769469 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 30041
I20240502 07:58:08.769604 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal/instance
I20240502 07:58:08.770177 30108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:08.775725 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f: Bootstrap starting.
I20240502 07:58:08.777760 30108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:08.778445 30108 log.cc:824] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:08.780090 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f: No bootstrap required, opened a new log
I20240502 07:58:08.782707 30108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } }
I20240502 07:58:08.782956 30108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:08.783015 30108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc31ce2ceb34fecae50f9a3fb34d84f, State: Initialized, Role: FOLLOWER
I20240502 07:58:08.783416 30108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } }
I20240502 07:58:08.783540 30108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:08.783630 30108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:08.783751 30108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:08.784663 30108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } }
I20240502 07:58:08.784976 30108 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc31ce2ceb34fecae50f9a3fb34d84f; no voters: 
I20240502 07:58:08.785240 30108 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 07:58:08.785616 30113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:08.785812 30113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [term 1 LEADER]: Becoming Leader. State: Replica: 1cc31ce2ceb34fecae50f9a3fb34d84f, State: Running, Role: LEADER
I20240502 07:58:08.786083 30113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } }
I20240502 07:58:08.786295 30108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:08.787068 30113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc31ce2ceb34fecae50f9a3fb34d84f. Latest consensus state: current_term: 1 leader_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } } }
I20240502 07:58:08.787266 30113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.787432 30114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc31ce2ceb34fecae50f9a3fb34d84f" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 45947 } } }
I20240502 07:58:08.787598 30114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.787652 30122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:08.790596 30122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:08.795950 30122 catalog_manager.cc:1347] Generated new cluster ID: 455f6b02842e4c7e8066e48ba6c3c1a1
I20240502 07:58:08.796078 30122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:08.806217 30122 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:08.807098 30122 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:08.811861 30122 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1cc31ce2ceb34fecae50f9a3fb34d84f: Generated new TSK 0
I20240502 07:58:08.812542 30122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:08.820778 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.65:0
--local_ip_for_outbound_sockets=127.29.83.65
--webserver_interface=127.29.83.65
--webserver_port=0
--tserver_master_addrs=127.29.83.126:45947
--builtin_ntp_servers=127.29.83.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 07:58:08.927016 30132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:08.927270 30132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:08.927335 30132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:08.930570 30132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:08.930673 30132 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 07:58:08.930689 30132 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 07:58:08.930711 30132 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 07:58:08.930759 30132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.65
I20240502 07:58:08.934710 30132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.29.83.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.29.83.126:45947
--never_fsync=true
--heap_profile_path=/tmp/kudu.30132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.65
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:08.935737 30132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.936975 30132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:08.946213 30137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.946830 30138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.947000 30140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.947682 30132 server_base.cc:1034] running on GCE node
I20240502 07:58:08.948118 30132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:08.948688 30132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:08.949864 30132 hybrid_clock.cc:648] HybridClock initialized: now 1714636688949840 us; error 45 us; skew 500 ppm
I20240502 07:58:08.951857 30132 webserver.cc:460] Webserver started at http://127.29.83.65:42937/ using document root <none> and password file <none>
I20240502 07:58:08.952432 30132 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.952497 30132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.952666 30132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.954301 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/instance:
uuid: "abeecd532ea6417fbb8200fd2c782145"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.954774 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/instance:
uuid: "abeecd532ea6417fbb8200fd2c782145"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.958127 30132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 07:58:08.960347 30146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:08.961338 30132 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:58:08.961445 30132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
uuid: "abeecd532ea6417fbb8200fd2c782145"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-2kd1"
I20240502 07:58:08.961535 30132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.976953 30132 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:08.977622 30132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:08.977790 30132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.978356 30132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:08.979291 30132 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:08.979341 30132 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:08.979403 30132 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:08.979431 30132 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:08.988433 30132 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.65:40393
I20240502 07:58:08.988457 30259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.65:40393 every 8 connection(s)
I20240502 07:58:08.989379 30132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
I20240502 07:58:08.996615 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 30132
I20240502 07:58:08.996724 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/instance
I20240502 07:58:08.998178 30260 heartbeater.cc:347] Connected to a master server at 127.29.83.126:45947
I20240502 07:58:08.998389 30260 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:08.998981 30260 heartbeater.cc:511] Master 127.29.83.126:45947 requested a full tablet report, sending...
I20240502 07:58:08.999212 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.66:0
--local_ip_for_outbound_sockets=127.29.83.66
--webserver_interface=127.29.83.66
--webserver_port=0
--tserver_master_addrs=127.29.83.126:45947
--builtin_ntp_servers=127.29.83.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 07:58:09.000432 30071 ts_manager.cc:194] Registered new tserver with Master: abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393)
I20240502 07:58:09.001578 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.65:33853
W20240502 07:58:09.110283 30263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.110529 30263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.110589 30263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.113802 30263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.113896 30263 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 07:58:09.113914 30263 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 07:58:09.113934 30263 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 07:58:09.113987 30263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.66
I20240502 07:58:09.117918 30263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.29.83.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.29.83.126:45947
--never_fsync=true
--heap_profile_path=/tmp/kudu.30263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.66
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.118899 30263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.120129 30263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.129097 30271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.129596 30268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.130313 30269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.131012 30263 server_base.cc:1034] running on GCE node
I20240502 07:58:09.131484 30263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.132133 30263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.133306 30263 hybrid_clock.cc:648] HybridClock initialized: now 1714636689133282 us; error 37 us; skew 500 ppm
I20240502 07:58:09.135313 30263 webserver.cc:460] Webserver started at http://127.29.83.66:46811/ using document root <none> and password file <none>
I20240502 07:58:09.135905 30263 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.136006 30263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.136197 30263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.137945 30263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/instance:
uuid: "133f2399e9b8473fbff0dfcab9a3d022"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.138419 30263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal/instance:
uuid: "133f2399e9b8473fbff0dfcab9a3d022"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.141747 30263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 07:58:09.143987 30277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.145129 30263 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:58:09.145246 30263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
uuid: "133f2399e9b8473fbff0dfcab9a3d022"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.145339 30263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.206848 30263 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:09.207564 30263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.207727 30263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.208348 30263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.209254 30263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.209301 30263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.209360 30263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.209393 30263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.217744 30263 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.66:39503
I20240502 07:58:09.217774 30390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.66:39503 every 8 connection(s)
I20240502 07:58:09.218715 30263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
I20240502 07:58:09.226135 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 30263
I20240502 07:58:09.226279 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal/instance
I20240502 07:58:09.228385 30391 heartbeater.cc:347] Connected to a master server at 127.29.83.126:45947
I20240502 07:58:09.228531 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.67:0
--local_ip_for_outbound_sockets=127.29.83.67
--webserver_interface=127.29.83.67
--webserver_port=0
--tserver_master_addrs=127.29.83.126:45947
--builtin_ntp_servers=127.29.83.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 07:58:09.228621 30391 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.229166 30391 heartbeater.cc:511] Master 127.29.83.126:45947 requested a full tablet report, sending...
I20240502 07:58:09.230145 30071 ts_manager.cc:194] Registered new tserver with Master: 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503)
I20240502 07:58:09.230859 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.66:55221
W20240502 07:58:09.339651 30394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.339890 30394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.339995 30394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.343218 30394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.343298 30394 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 07:58:09.343314 30394 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 07:58:09.343335 30394 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 07:58:09.343379 30394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.67
I20240502 07:58:09.347102 30394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.29.83.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.29.83.126:45947
--never_fsync=true
--heap_profile_path=/tmp/kudu.30394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.67
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.348052 30394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.349237 30394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.357435 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.358052 30400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.358789 30399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.359437 30394 server_base.cc:1034] running on GCE node
I20240502 07:58:09.359850 30394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.360471 30394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.361634 30394 hybrid_clock.cc:648] HybridClock initialized: now 1714636689361620 us; error 31 us; skew 500 ppm
I20240502 07:58:09.363545 30394 webserver.cc:460] Webserver started at http://127.29.83.67:33395/ using document root <none> and password file <none>
I20240502 07:58:09.364210 30394 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.364284 30394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.364480 30394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.366210 30394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/instance:
uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.366677 30394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal/instance:
uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.370093 30394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20240502 07:58:09.372450 30408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.373755 30394 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:58:09.373868 30394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-2kd1"
I20240502 07:58:09.373988 30394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.400214 30394 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:09.400857 30394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.400995 30394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.401495 30394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.402366 30394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.402415 30394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.402454 30394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.402474 30394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:09.410727 30394 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.67:38759
I20240502 07:58:09.410761 30521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.67:38759 every 8 connection(s)
I20240502 07:58:09.411631 30394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
I20240502 07:58:09.414722 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 30394
I20240502 07:58:09.414814 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal/instance
I20240502 07:58:09.420022 30522 heartbeater.cc:347] Connected to a master server at 127.29.83.126:45947
I20240502 07:58:09.420236 30522 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.420715 30522 heartbeater.cc:511] Master 127.29.83.126:45947 requested a full tablet report, sending...
I20240502 07:58:09.421622 30071 ts_manager.cc:194] Registered new tserver with Master: 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.422169 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.67:36997
I20240502 07:58:09.429152 30029 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 07:58:09.440522 30071 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36400:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240502 07:58:09.443755 30071 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:58:09.460788 30325 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2cd669a2b4b06a268464e4769d8db (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 07:58:09.460788 30192 tablet_service.cc:1467] Processing CreateTablet for tablet 096d782a2b994f25a0269a0e6f15bbbd (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 07:58:09.461431 30454 tablet_service.cc:1467] Processing CreateTablet for tablet 096d782a2b994f25a0269a0e6f15bbbd (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 07:58:09.461431 30455 tablet_service.cc:1467] Processing CreateTablet for tablet 4128a446093e425a8fa63ef6d0ed7224 (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 07:58:09.462056 30325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2cd669a2b4b06a268464e4769d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.462057 30192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096d782a2b994f25a0269a0e6f15bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.462551 30455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128a446093e425a8fa63ef6d0ed7224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.462846 30454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096d782a2b994f25a0269a0e6f15bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.461656 30456 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2cd669a2b4b06a268464e4769d8db (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 07:58:09.463587 30456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2cd669a2b4b06a268464e4769d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.464195 30323 tablet_service.cc:1467] Processing CreateTablet for tablet 096d782a2b994f25a0269a0e6f15bbbd (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 07:58:09.464277 30324 tablet_service.cc:1467] Processing CreateTablet for tablet 4128a446093e425a8fa63ef6d0ed7224 (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 07:58:09.464464 30323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096d782a2b994f25a0269a0e6f15bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.465102 30324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128a446093e425a8fa63ef6d0ed7224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.468086 30194 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2cd669a2b4b06a268464e4769d8db (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 07:58:09.468379 30194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2cd669a2b4b06a268464e4769d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.470563 30541 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: Bootstrap starting.
I20240502 07:58:09.471125 30193 tablet_service.cc:1467] Processing CreateTablet for tablet 4128a446093e425a8fa63ef6d0ed7224 (DEFAULT_TABLE table=TestTable [id=480a9f5307754a1e978132e6c71c75c9]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 07:58:09.471388 30193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128a446093e425a8fa63ef6d0ed7224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:09.472800 30541 tablet_bootstrap.cc:654] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.472954 30539 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Bootstrap starting.
I20240502 07:58:09.473925 30541 log.cc:824] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:09.474601 30540 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: Bootstrap starting.
I20240502 07:58:09.474962 30539 tablet_bootstrap.cc:654] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.475505 30541 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: No bootstrap required, opened a new log
I20240502 07:58:09.475697 30541 ts_tablet_manager.cc:1367] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 07:58:09.475981 30539 log.cc:824] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:09.476620 30540 tablet_bootstrap.cc:654] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.477345 30540 log.cc:824] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:09.477739 30539 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: No bootstrap required, opened a new log
I20240502 07:58:09.477881 30539 ts_tablet_manager.cc:1367] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 07:58:09.478683 30541 raft_consensus.cc:348] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.478837 30541 raft_consensus.cc:374] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.478899 30541 raft_consensus.cc:729] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.478940 30540 tablet_bootstrap.cc:492] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: No bootstrap required, opened a new log
I20240502 07:58:09.479091 30540 ts_tablet_manager.cc:1367] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 07:58:09.479264 30541 consensus_queue.cc:260] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.479820 30391 heartbeater.cc:503] Master 127.29.83.126:45947 was elected leader, sending a full tablet report...
I20240502 07:58:09.481101 30541 ts_tablet_manager.cc:1398] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 07:58:09.481241 30539 raft_consensus.cc:348] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.481406 30541 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: Bootstrap starting.
I20240502 07:58:09.481475 30539 raft_consensus.cc:374] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.481559 30539 raft_consensus.cc:729] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.482089 30539 consensus_queue.cc:260] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.482398 30540 raft_consensus.cc:348] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.482583 30540 raft_consensus.cc:374] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.482640 30540 raft_consensus.cc:729] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfdd8e06ad74aa48ae8115d0ad2f473, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.482671 30541 tablet_bootstrap.cc:654] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.483134 30539 ts_tablet_manager.cc:1398] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 07:58:09.483093 30540 consensus_queue.cc:260] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.483542 30260 heartbeater.cc:503] Master 127.29.83.126:45947 was elected leader, sending a full tablet report...
I20240502 07:58:09.483872 30540 ts_tablet_manager.cc:1398] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 07:58:09.484622 30539 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: Bootstrap starting.
I20240502 07:58:09.484772 30522 heartbeater.cc:503] Master 127.29.83.126:45947 was elected leader, sending a full tablet report...
I20240502 07:58:09.485669 30540 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: Bootstrap starting.
I20240502 07:58:09.485829 30539 tablet_bootstrap.cc:654] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.486701 30540 tablet_bootstrap.cc:654] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.487075 30541 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: No bootstrap required, opened a new log
I20240502 07:58:09.487074 30539 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: No bootstrap required, opened a new log
I20240502 07:58:09.487154 30539 ts_tablet_manager.cc:1367] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 07:58:09.487154 30541 ts_tablet_manager.cc:1367] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 07:58:09.487720 30539 raft_consensus.cc:348] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.487779 30541 raft_consensus.cc:348] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.487903 30541 raft_consensus.cc:374] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.487957 30539 raft_consensus.cc:374] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.488001 30541 raft_consensus.cc:729] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.488010 30539 raft_consensus.cc:729] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.488126 30541 consensus_queue.cc:260] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.488126 30539 consensus_queue.cc:260] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.488392 30541 ts_tablet_manager.cc:1398] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 07:58:09.488472 30539 ts_tablet_manager.cc:1398] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240502 07:58:09.488538 30541 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022: Bootstrap starting.
I20240502 07:58:09.488600 30539 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145: Bootstrap starting.
I20240502 07:58:09.489446 30539 tablet_bootstrap.cc:654] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.489446 30541 tablet_bootstrap.cc:654] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.489825 30540 tablet_bootstrap.cc:492] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: No bootstrap required, opened a new log
I20240502 07:58:09.489935 30540 ts_tablet_manager.cc:1367] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
W20240502 07:58:09.490351 30261 tablet.cc:2369] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:09.490607 30540 raft_consensus.cc:348] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.490706 30540 raft_consensus.cc:374] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.490744 30540 raft_consensus.cc:729] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfdd8e06ad74aa48ae8115d0ad2f473, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.490841 30540 consensus_queue.cc:260] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.491104 30540 ts_tablet_manager.cc:1398] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 07:58:09.491254 30540 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473: Bootstrap starting.
I20240502 07:58:09.491761 30539 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145: No bootstrap required, opened a new log
I20240502 07:58:09.491884 30539 ts_tablet_manager.cc:1367] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 07:58:09.491935 30541 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022: No bootstrap required, opened a new log
I20240502 07:58:09.492007 30541 ts_tablet_manager.cc:1367] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 07:58:09.492189 30540 tablet_bootstrap.cc:654] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:09.492445 30541 raft_consensus.cc:348] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.492554 30541 raft_consensus.cc:374] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.492592 30541 raft_consensus.cc:729] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.492470 30539 raft_consensus.cc:348] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.492691 30539 raft_consensus.cc:374] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.492769 30539 raft_consensus.cc:729] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.492996 30539 consensus_queue.cc:260] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.493008 30541 consensus_queue.cc:260] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.493278 30541 ts_tablet_manager.cc:1398] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 07:58:09.493292 30540 tablet_bootstrap.cc:492] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473: No bootstrap required, opened a new log
I20240502 07:58:09.493362 30540 ts_tablet_manager.cc:1367] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 07:58:09.493584 30539 ts_tablet_manager.cc:1398] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 07:58:09.493773 30540 raft_consensus.cc:348] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.493870 30540 raft_consensus.cc:374] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:09.493913 30540 raft_consensus.cc:729] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfdd8e06ad74aa48ae8115d0ad2f473, State: Initialized, Role: FOLLOWER
I20240502 07:58:09.494017 30540 consensus_queue.cc:260] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.494308 30540 ts_tablet_manager.cc:1398] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 07:58:09.508672 30545 raft_consensus.cc:482] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:09.508852 30545 raft_consensus.cc:504] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.510018 30545 leader_election.cc:288] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.513904 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145" is_pre_election: true
I20240502 07:58:09.514295 30214 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.514653 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:09.514734 30279 leader_election.cc:302] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:09.514881 30476 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.515026 30545 raft_consensus.cc:2781] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:09.515101 30545 raft_consensus.cc:482] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:09.515146 30545 raft_consensus.cc:3037] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.516125 30545 raft_consensus.cc:504] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.516459 30545 leader_election.cc:288] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Requested vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.516789 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145"
I20240502 07:58:09.516856 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:09.516909 30214 raft_consensus.cc:3037] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.516978 30476 raft_consensus.cc:3037] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.517830 30476 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.518078 30214 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.518119 30279 leader_election.cc:302] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, 6cfdd8e06ad74aa48ae8115d0ad2f473; no voters: 
I20240502 07:58:09.518292 30545 raft_consensus.cc:2781] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:09.518496 30545 raft_consensus.cc:686] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 LEADER]: Becoming Leader. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Running, Role: LEADER
I20240502 07:58:09.518786 30545 consensus_queue.cc:237] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.521900 30071 catalog_manager.cc:5484] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66). New cstate: current_term: 1 leader_uuid: "133f2399e9b8473fbff0dfcab9a3d022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:09.539759 30545 raft_consensus.cc:482] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:09.539889 30545 raft_consensus.cc:504] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.540340 30545 leader_election.cc:288] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.540838 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:09.540838 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145" is_pre_election: true
I20240502 07:58:09.540992 30214 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.540992 30476 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.541342 30279 leader_election.cc:302] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:09.541564 30545 raft_consensus.cc:2781] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:09.541637 30545 raft_consensus.cc:482] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:09.541669 30545 raft_consensus.cc:3037] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.542675 30545 raft_consensus.cc:504] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.543223 30545 leader_election.cc:288] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Requested vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.543641 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145"
I20240502 07:58:09.543695 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:09.543754 30214 raft_consensus.cc:3037] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.543825 30476 raft_consensus.cc:3037] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.544627 30214 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.544793 30476 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.544947 30279 leader_election.cc:302] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:09.545101 30545 raft_consensus.cc:2781] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:09.545212 30545 raft_consensus.cc:686] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 LEADER]: Becoming Leader. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Running, Role: LEADER
I20240502 07:58:09.545367 30545 consensus_queue.cc:237] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.547317 30071 catalog_manager.cc:5484] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66). New cstate: current_term: 1 leader_uuid: "133f2399e9b8473fbff0dfcab9a3d022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:09.555212 30545 raft_consensus.cc:482] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:09.555346 30545 raft_consensus.cc:504] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.555734 30545 leader_election.cc:288] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.556236 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145" is_pre_election: true
I20240502 07:58:09.556267 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:09.556378 30214 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.556401 30476 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 0.
I20240502 07:58:09.556726 30279 leader_election.cc:302] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:09.556903 30545 raft_consensus.cc:2781] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:09.556960 30545 raft_consensus.cc:482] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:09.556998 30545 raft_consensus.cc:3037] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.557920 30545 raft_consensus.cc:504] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.558269 30545 leader_election.cc:288] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Requested vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:09.558694 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:09.558676 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "133f2399e9b8473fbff0dfcab9a3d022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145"
I20240502 07:58:09.558809 30476 raft_consensus.cc:3037] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.558809 30214 raft_consensus.cc:3037] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:09.559723 30214 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.559723 30476 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133f2399e9b8473fbff0dfcab9a3d022 in term 1.
I20240502 07:58:09.560056 30279 leader_election.cc:302] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133f2399e9b8473fbff0dfcab9a3d022, 6cfdd8e06ad74aa48ae8115d0ad2f473; no voters: 
I20240502 07:58:09.560225 30545 raft_consensus.cc:2781] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:09.560313 30545 raft_consensus.cc:686] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [term 1 LEADER]: Becoming Leader. State: Replica: 133f2399e9b8473fbff0dfcab9a3d022, State: Running, Role: LEADER
I20240502 07:58:09.560456 30545 consensus_queue.cc:237] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:09.562072 30071 catalog_manager.cc:5484] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66). New cstate: current_term: 1 leader_uuid: "133f2399e9b8473fbff0dfcab9a3d022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:09.583765 30029 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 07:58:09.585868 30029 ts_itest-base.cc:233] Waiting for 3 tablets on tserver abeecd532ea6417fbb8200fd2c782145 to finish bootstrapping
I20240502 07:58:09.592072 30029 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 133f2399e9b8473fbff0dfcab9a3d022 to finish bootstrapping
I20240502 07:58:09.597543 30029 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 6cfdd8e06ad74aa48ae8115d0ad2f473 to finish bootstrapping
I20240502 07:58:09.603652 30560 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
  http://127.29.83.67:33395/mem-trackers
  http://127.29.83.67:33395/stacks
  http://127.29.83.65:42937/transactions?tablet_id=096d782a2b994f25a0269a0e6f15bbbd
  http://127.29.83.66:46811/threadz?group=all
  http://127.29.83.126:38175/tables
  http://127.29.83.126:38175/ipki-ca-cert
  http://127.29.83.66:46811/threadz
  http://127.29.83.65:42937/startup
  http://127.29.83.67:33395/scans
  http://127.29.83.67:33395/tablets
  http://127.29.83.66:46811/maintenance-manager
  http://127.29.83.66:46811/tablets
  http://127.29.83.65:42937/mem-trackers
  http://127.29.83.126:38175/dump-entities
  http://127.29.83.66:46811/startup
  http://127.29.83.126:38175/startup
  http://127.29.83.126:38175/tablet-servers
  http://127.29.83.65:42937/threadz?group=all
  http://127.29.83.126:38175/threadz
  http://127.29.83.65:42937/maintenance-manager
  http://127.29.83.65:42937/threadz
  http://127.29.83.126:38175/ipki-ca-cert-pem
  http://127.29.83.66:46811/scans
  http://127.29.83.65:42937/scans
  http://127.29.83.67:33395/threadz
  http://127.29.83.66:46811/transactions?tablet_id=096d782a2b994f25a0269a0e6f15bbbd
  http://127.29.83.65:42937/metrics
  http://127.29.83.65:42937/stacks
  http://127.29.83.67:33395/startup
  http://127.29.83.66:46811/metrics
  http://127.29.83.66:46811/stacks
  http://127.29.83.67:33395/transactions?tablet_id=096d782a2b994f25a0269a0e6f15bbbd
  http://127.29.83.126:38175/mem-trackers
  http://127.29.83.126:38175/stacks
  http://127.29.83.67:33395/threadz?group=all
  http://127.29.83.126:38175/masters
  http://127.29.83.67:33395/metrics
  http://127.29.83.65:42937/tablets
  http://127.29.83.126:38175/metrics
  http://127.29.83.126:38175/threadz?group=all
  http://127.29.83.67:33395/maintenance-manager
  http://127.29.83.66:46811/mem-trackers
  http://127.29.83.126:38175/ipki-ca-cert-der
I20240502 07:58:09.606597 30029 linked_list-test-util.h:401] Running for: 30.000s
I20240502 07:58:09.606680 30029 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 07:58:09.606705 30029 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240502 07:58:09.641803 30476 raft_consensus.cc:1261] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.642004 30214 raft_consensus.cc:1261] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.642167 30212 raft_consensus.cc:1261] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.642010 30213 raft_consensus.cc:1261] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.642903 30476 raft_consensus.cc:1261] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.642894 30567 consensus_queue.cc:1035] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.642918 30475 raft_consensus.cc:1261] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Refusing update from remote peer 133f2399e9b8473fbff0dfcab9a3d022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:09.643666 30567 consensus_queue.cc:1035] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.643721 30568 consensus_queue.cc:1035] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.643895 30545 consensus_queue.cc:1035] T 096d782a2b994f25a0269a0e6f15bbbd P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.644248 30567 consensus_queue.cc:1035] T 4128a446093e425a8fa63ef6d0ed7224 P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.644376 30545 consensus_queue.cc:1035] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:09.658087 30575 mvcc.cc:205] Tried to move back new op lower bound from 7023151880765562880 to 7023151880379617280. Current Snapshot: MvccSnapshot[applied={T|T < 7023151880765562880}]
I20240502 07:58:09.659348 30564 mvcc.cc:205] Tried to move back new op lower bound from 7023151880764612608 to 7023151880440815616. Current Snapshot: MvccSnapshot[applied={T|T < 7023151880764612608}]
I20240502 07:58:09.661664 30579 mvcc.cc:205] Tried to move back new op lower bound from 7023151880765562880 to 7023151880379617280. Current Snapshot: MvccSnapshot[applied={T|T < 7023151880765562880}]
I20240502 07:58:10.250877 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:10.308300 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:10.309660 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:12.971503 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:13.018373 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:13.019728 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:13.421020 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:13.476444 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:13.503643 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
W20240502 07:58:14.262377 30413 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.335s	user 0.001s	sys 0.000s
I20240502 07:58:14.264971 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.843s	user 0.285s	sys 0.005s Metrics: {"bytes_written":150380,"cfile_init":1,"compiler_manager_pool.queue_time_us":5235,"dirs.queue_time_us":24123,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2657,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3087,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8192,"thread_start_us":208,"threads_started":2}
I20240502 07:58:14.266635 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 133045 bytes on disk
I20240502 07:58:14.267549 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 07:58:14.316169 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.839s	user 0.296s	sys 0.008s Metrics: {"bytes_written":10103,"cfile_init":1,"compiler_manager_pool.queue_time_us":14764,"delete_count":0,"dirs.queue_time_us":20039,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":5495,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1592,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8291,"thread_start_us":183,"threads_started":2,"update_count":1553}
I20240502 07:58:14.317500 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.076765
I20240502 07:58:14.376441 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.058s	user 0.038s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":11306,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":66,"lbm_read_time_us":341,"lbm_reads_lt_1ms":24,"lbm_write_time_us":255,"lbm_writes_lt_1ms":8,"mutex_wait_us":2,"reinsert_count":0,"thread_start_us":89,"threads_started":1,"update_count":1553}
I20240502 07:58:14.377102 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 134140 bytes on disk
I20240502 07:58:14.378698 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 07:58:14.483757 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.980s	user 0.330s	sys 0.000s Metrics: {"bytes_written":10553,"cfile_init":1,"compiler_manager_pool.queue_time_us":114723,"compiler_manager_pool.run_cpu_time_us":286219,"compiler_manager_pool.run_wall_time_us":888622,"delete_count":0,"dirs.queue_time_us":18215,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1952,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1723,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8363,"thread_start_us":200,"threads_started":2,"update_count":1625}
I20240502 07:58:14.485088 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.079529
I20240502 07:58:14.521533 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.036s	user 0.035s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":11753,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":51,"lbm_read_time_us":251,"lbm_reads_lt_1ms":24,"lbm_write_time_us":192,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":106,"threads_started":1,"update_count":1625}
I20240502 07:58:14.522783 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 134750 bytes on disk
I20240502 07:58:14.523557 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240502 07:58:16.563043 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:16.597358 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:16.598565 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:17.523543 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:17.636492 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:17.779662 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:17.858414 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.335s	user 0.222s	sys 0.004s Metrics: {"bytes_written":144116,"cfile_init":1,"dirs.queue_time_us":17127,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1447,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7834,"thread_start_us":82,"threads_started":1}
I20240502 07:58:17.864490 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 128257 bytes on disk
I20240502 07:58:17.865003 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 07:58:17.866966 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014822
I20240502 07:58:18.164093 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.384s	user 0.294s	sys 0.008s Metrics: {"bytes_written":146431,"cfile_init":1,"dirs.queue_time_us":11460,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1766,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7986,"thread_start_us":88,"threads_started":1}
I20240502 07:58:18.165551 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 128433 bytes on disk
I20240502 07:58:18.166167 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 07:58:18.166880 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014818
I20240502 07:58:18.370819 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.734s	user 0.242s	sys 0.004s Metrics: {"bytes_written":9388,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":8102,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":87000,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4532,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7929,"thread_start_us":76,"threads_started":1,"update_count":1442}
I20240502 07:58:18.372067 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 128302 bytes on disk
I20240502 07:58:18.372577 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 07:58:18.373162 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014817
I20240502 07:58:18.963304 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.590s	user 0.426s	sys 0.000s Metrics: {"bytes_written":295629,"cfile_cache_hit":12,"cfile_cache_hit_bytes":653,"cfile_cache_miss":64,"cfile_cache_miss_bytes":1071915,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":2037,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":2656,"drs_written":1,"lbm_read_time_us":4078,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":99,"lbm_write_time_us":2825,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":533184,"rows_written":16220,"thread_start_us":554,"threads_started":8}
I20240502 07:58:18.964161 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 295425 bytes on disk
I20240502 07:58:18.972433 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 07:58:19.325502 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.158s	user 0.509s	sys 0.015s Metrics: {"bytes_written":10183,"cfile_cache_hit":14,"cfile_cache_hit_bytes":145821,"cfile_cache_miss":58,"cfile_cache_miss_bytes":914257,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1678,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":87997,"drs_written":1,"lbm_read_time_us":3022,"lbm_reads_lt_1ms":90,"lbm_write_time_us":3033,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":509322,"reinsert_count":0,"rows_written":16349,"thread_start_us":449,"threads_started":7,"update_count":1571}
I20240502 07:58:19.326256 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 284631 bytes on disk
I20240502 07:58:19.326781 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 07:58:19.327237 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.039345
I20240502 07:58:19.372390 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.505s	user 0.430s	sys 0.008s Metrics: {"bytes_written":8128,"cfile_cache_hit":13,"cfile_cache_hit_bytes":142673,"cfile_cache_miss":55,"cfile_cache_miss_bytes":884727,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":10960,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":4922,"drs_written":1,"lbm_read_time_us":5616,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":82,"lbm_write_time_us":2770,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":478193,"reinsert_count":0,"rows_written":16026,"thread_start_us":1870,"threads_started":8,"update_count":1248}
I20240502 07:58:19.373538 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.032052
I20240502 07:58:19.411509 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.082s	user 0.046s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":12522,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":279,"lbm_reads_lt_1ms":20,"lbm_write_time_us":249,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1571}
I20240502 07:58:19.429286 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.055s	user 0.039s	sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":10299,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":318,"lbm_reads_lt_1ms":24,"lbm_write_time_us":254,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1248}
I20240502 07:58:19.431193 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 281406 bytes on disk
I20240502 07:58:19.432449 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":252,"lbm_reads_lt_1ms":4}
I20240502 07:58:19.615463 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023151921598672896 Inserted count: 28000
I20240502 07:58:19.775020 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:19.812505 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:19.813783 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:22.229876 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:22.356570 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:22.386124 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:22.387298 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:22.421799 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:22.440297 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:22.545845 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.316s	user 0.224s	sys 0.000s Metrics: {"bytes_written":144021,"cfile_init":1,"dirs.queue_time_us":13720,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1399,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7828,"spinlock_wait_cycles":217344,"thread_start_us":92,"threads_started":1}
I20240502 07:58:22.546844 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 137940 bytes on disk
I20240502 07:58:22.549461 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 07:58:22.550132 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014707
I20240502 07:58:22.828017 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.405s	user 0.230s	sys 0.000s Metrics: {"bytes_written":146700,"cfile_init":1,"dirs.queue_time_us":14589,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1110,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1691,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7961,"thread_start_us":97,"threads_started":1}
I20240502 07:58:22.830651 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling LogGCOp(ddd2cd669a2b4b06a268464e4769d8db): free 1053216 bytes of WAL
I20240502 07:58:22.832226 30151 log_reader.cc:386] T ddd2cd669a2b4b06a268464e4769d8db: removed 1 log segments from log reader
I20240502 07:58:22.832320 30151 log.cc:1076] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145: Deleting log segment in path: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/wals/ddd2cd669a2b4b06a268464e4769d8db/wal-000000001 (ops 1-733)
I20240502 07:58:22.832903 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: LogGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240502 07:58:22.833453 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 138223 bytes on disk
I20240502 07:58:22.834306 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 07:58:22.835041 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014705
I20240502 07:58:23.520292 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.080s	user 0.250s	sys 0.000s Metrics: {"bytes_written":152213,"cfile_init":1,"dirs.queue_time_us":1222,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":9551,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5177,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8317,"thread_start_us":70,"threads_started":1}
I20240502 07:58:23.520987 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling LogGCOp(ddd2cd669a2b4b06a268464e4769d8db): free 1052880 bytes of WAL
I20240502 07:58:23.521201 30413 log_reader.cc:386] T ddd2cd669a2b4b06a268464e4769d8db: removed 1 log segments from log reader
I20240502 07:58:23.521252 30413 log.cc:1076] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473: Deleting log segment in path: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal/wals/ddd2cd669a2b4b06a268464e4769d8db/wal-000000001 (ops 1-733)
I20240502 07:58:23.521740 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: LogGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240502 07:58:23.522044 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014703
I20240502 07:58:23.848516 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.013s	user 0.708s	sys 0.012s Metrics: {"bytes_written":2118,"cfile_cache_hit":12,"cfile_cache_hit_bytes":47632,"cfile_cache_miss":76,"cfile_cache_miss_bytes":1581558,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":21677,"drs_written":1,"lbm_read_time_us":8274,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":107,"lbm_write_time_us":3660,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":798832,"reinsert_count":0,"rows_written":24310,"thread_start_us":614,"threads_started":8,"update_count":298}
I20240502 07:58:23.849321 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.005569
I20240502 07:58:23.860982 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.311s	user 0.612s	sys 0.004s Metrics: {"bytes_written":414,"cfile_cache_hit":11,"cfile_cache_hit_bytes":46912,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1573740,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":8845,"drs_written":1,"lbm_read_time_us":2827,"lbm_reads_lt_1ms":103,"lbm_write_time_us":13072,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":66,"num_input_rowsets":2,"peak_mem_usage":794780,"reinsert_count":0,"rows_written":24048,"thread_start_us":626,"threads_started":8,"update_count":36}
I20240502 07:58:23.862108 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.001103
I20240502 07:58:23.897805 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.048s	user 0.039s	sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3193,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":63,"lbm_read_time_us":338,"lbm_reads_lt_1ms":24,"lbm_write_time_us":493,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":298}
I20240502 07:58:23.898499 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 446480 bytes on disk
I20240502 07:58:23.899396 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 07:58:23.945013 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.080s	user 0.044s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":532,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":757,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":623,"lbm_read_time_us":364,"lbm_reads_lt_1ms":24,"lbm_write_time_us":249,"lbm_writes_lt_1ms":8,"mutex_wait_us":700,"reinsert_count":0,"update_count":36}
I20240502 07:58:23.946702 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 443669 bytes on disk
I20240502 07:58:23.948256 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 07:58:24.658603 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.136s	user 0.862s	sys 0.007s Metrics: {"bytes_written":441536,"cfile_cache_hit":14,"cfile_cache_hit_bytes":190663,"cfile_cache_miss":76,"cfile_cache_miss_bytes":1441883,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":10432,"dirs.run_cpu_time_us":359,"dirs.run_wall_time_us":3242,"drs_written":1,"lbm_read_time_us":2903,"lbm_reads_lt_1ms":112,"lbm_write_time_us":3767,"lbm_writes_lt_1ms":64,"mutex_wait_us":407,"num_input_rowsets":2,"peak_mem_usage":799341,"rows_written":24343,"thread_start_us":682,"threads_started":7}
I20240502 07:58:24.659432 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 449671 bytes on disk
I20240502 07:58:24.659873 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 07:58:25.065410 30561 thread.cc:406] Could not get per-thread statistics: Not found: /proc/self/task/30626/stat: No such file or directory (error 2)
I20240502 07:58:25.870234 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:25.916803 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:25.917793 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:26.155611 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224): perf score=1.000000
I20240502 07:58:26.163715 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224): perf score=1.000000
I20240502 07:58:26.203176 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224): perf score=1.000000
I20240502 07:58:26.549348 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.394s	user 0.226s	sys 0.013s Metrics: {"bytes_written":3376,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15203,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":9962,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1684,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7748,"thread_start_us":85,"threads_started":1,"update_count":497}
I20240502 07:58:26.553437 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd): perf score=1.000000
I20240502 07:58:26.566890 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.363s	user 0.235s	sys 0.000s Metrics: {"bytes_written":3512,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13861,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":797,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1581,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7769,"thread_start_us":75,"threads_started":1,"update_count":518}
I20240502 07:58:26.569793 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd): perf score=1.000000
I20240502 07:58:26.692978 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.529s	user 0.243s	sys 0.000s Metrics: {"bytes_written":3376,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11057,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":6743,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1741,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7748,"thread_start_us":83,"threads_started":1,"update_count":497}
I20240502 07:58:26.693708 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd): perf score=1.000000
I20240502 07:58:26.820694 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.251s	user 0.223s	sys 0.000s Metrics: {"bytes_written":143027,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":917,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1555,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7810}
I20240502 07:58:26.821589 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224): perf score=0.028698
I20240502 07:58:26.846793 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.025s	user 0.022s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4490,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":32,"lbm_read_time_us":1095,"lbm_reads_lt_1ms":24,"lbm_write_time_us":258,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":67,"threads_started":1,"update_count":518}
I20240502 07:58:26.847695 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd): 144716 bytes on disk
I20240502 07:58:26.848188 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 07:58:26.848806 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224): 140544 bytes on disk
I20240502 07:58:26.849212 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 07:58:26.849687 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:26.888478 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.332s	user 0.223s	sys 0.008s Metrics: {"bytes_written":142855,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":710,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1764,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7802}
I20240502 07:58:26.889371 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd): 144646 bytes on disk
I20240502 07:58:26.891062 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 07:58:26.891659 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224): perf score=0.027674
I20240502 07:58:26.936856 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.044s	user 0.021s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4333,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":43,"lbm_read_time_us":291,"lbm_reads_lt_1ms":24,"lbm_write_time_us":213,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":85,"threads_started":1,"update_count":497}
I20240502 07:58:26.937583 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224): 140354 bytes on disk
I20240502 07:58:26.937961 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 07:58:27.067781 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.218s	user 0.212s	sys 0.000s Metrics: {"bytes_written":143119,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":749,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1453,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7765}
I20240502 07:58:27.068658 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling LogGCOp(ddd2cd669a2b4b06a268464e4769d8db): free 1055228 bytes of WAL
I20240502 07:58:27.069300 30282 log_reader.cc:386] T ddd2cd669a2b4b06a268464e4769d8db: removed 1 log segments from log reader
I20240502 07:58:27.069422 30282 log.cc:1076] T ddd2cd669a2b4b06a268464e4769d8db P 133f2399e9b8473fbff0dfcab9a3d022: Deleting log segment in path: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal/wals/ddd2cd669a2b4b06a268464e4769d8db/wal-000000001 (ops 1-734)
I20240502 07:58:27.069927 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: LogGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240502 07:58:27.070632 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 130658 bytes on disk
I20240502 07:58:27.071048 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 07:58:27.072221 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014578
I20240502 07:58:27.188977 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:27.203981 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.503s	user 0.323s	sys 0.004s Metrics: {"bytes_written":143991,"cfile_init":1,"dirs.queue_time_us":9636,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1685,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7870,"thread_start_us":104,"threads_started":1}
I20240502 07:58:27.204741 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd): 145390 bytes on disk
I20240502 07:58:27.212291 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(096d782a2b994f25a0269a0e6f15bbbd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 07:58:27.212952 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224): 140354 bytes on disk
I20240502 07:58:27.213290 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 07:58:27.213789 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224): perf score=0.027674
I20240502 07:58:27.248113 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: MajorDeltaCompactionOp(4128a446093e425a8fa63ef6d0ed7224) complete. Timing: real 0.034s	user 0.023s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4333,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":38,"lbm_read_time_us":278,"lbm_reads_lt_1ms":20,"lbm_write_time_us":289,"lbm_writes_lt_1ms":8,"mutex_wait_us":166,"reinsert_count":0,"thread_start_us":86,"threads_started":1,"update_count":497}
I20240502 07:58:27.248785 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:27.670467 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.420s	user 0.250s	sys 0.004s Metrics: {"bytes_written":7790,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":779,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1648,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7834,"update_count":1189}
I20240502 07:58:27.671309 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 129024 bytes on disk
I20240502 07:58:27.681243 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 07:58:27.682646 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014575
I20240502 07:58:27.802414 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.610s	user 0.243s	sys 0.008s Metrics: {"bytes_written":7582,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12869,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":36295,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7834,"thread_start_us":99,"threads_started":1,"update_count":1156}
I20240502 07:58:27.803443 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014575
I20240502 07:58:28.361745 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.289s	user 0.757s	sys 0.000s Metrics: {"bytes_written":5466,"cfile_cache_hit":12,"cfile_cache_hit_bytes":46753,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2096103,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":62283,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":33981,"drs_written":1,"lbm_read_time_us":3376,"lbm_reads_lt_1ms":126,"lbm_write_time_us":4485,"lbm_writes_lt_1ms":79,"num_input_rowsets":2,"peak_mem_usage":1040742,"reinsert_count":0,"rows_written":31813,"thread_start_us":606,"threads_started":8,"update_count":825}
I20240502 07:58:28.363487 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 585979 bytes on disk
I20240502 07:58:28.364127 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 07:58:28.364714 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.010993
I20240502 07:58:28.512475 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.148s	user 0.058s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7999,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":127,"lbm_read_time_us":253,"lbm_reads_lt_1ms":20,"lbm_write_time_us":207,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":825}
I20240502 07:58:29.029265 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.225s	user 0.866s	sys 0.003s Metrics: {"bytes_written":577915,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1042,"cfile_cache_miss":98,"cfile_cache_miss_bytes":2173600,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":22212,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":10353,"drs_written":1,"lbm_read_time_us":12503,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":137,"lbm_write_time_us":13960,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":75,"mutex_wait_us":25,"num_input_rowsets":2,"peak_mem_usage":1066592,"rows_written":32144,"thread_start_us":471,"threads_started":8}
I20240502 07:58:29.029955 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 599321 bytes on disk
I20240502 07:58:29.030325 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 07:58:29.123349 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.440s	user 0.792s	sys 0.004s Metrics: {"bytes_written":578530,"cfile_cache_hit":13,"cfile_cache_hit_bytes":277560,"cfile_cache_miss":93,"cfile_cache_miss_bytes":1900120,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":14856,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":5971,"drs_written":1,"lbm_read_time_us":3405,"lbm_reads_lt_1ms":125,"lbm_write_time_us":4367,"lbm_writes_lt_1ms":76,"mutex_wait_us":26,"num_input_rowsets":2,"peak_mem_usage":1067729,"rows_written":32177,"spinlock_wait_cycles":2392704,"thread_start_us":551,"threads_started":8}
I20240502 07:58:29.124326 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 599878 bytes on disk
I20240502 07:58:29.125140 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 07:58:29.614486 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023151962538520576 Inserted count: 53700
I20240502 07:58:30.247617 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:30.286610 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:30.287693 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:31.769276 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:31.786809 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:31.880920 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:32.047849 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.278s	user 0.244s	sys 0.000s Metrics: {"bytes_written":151680,"cfile_init":1,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1585,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8282,"thread_start_us":114,"threads_started":1}
I20240502 07:58:32.049149 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014463
I20240502 07:58:32.146816 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.360s	user 0.223s	sys 0.000s Metrics: {"bytes_written":146918,"cfile_init":1,"dirs.queue_time_us":10930,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":8778,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1505,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7984,"thread_start_us":75,"threads_started":1}
I20240502 07:58:32.147840 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 137354 bytes on disk
I20240502 07:58:32.148350 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 07:58:32.148905 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014461
W20240502 07:58:33.261950 30413 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.529s	user 0.001s	sys 0.000s
I20240502 07:58:33.263617 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.382s	user 0.237s	sys 0.000s Metrics: {"bytes_written":148378,"cfile_init":1,"dirs.queue_time_us":16232,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":7994,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1493,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8049,"thread_start_us":76,"threads_started":1}
I20240502 07:58:33.264360 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 137649 bytes on disk
I20240502 07:58:33.265866 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 07:58:33.266438 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014460
I20240502 07:58:33.295547 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.146s	user 0.970s	sys 0.014s Metrics: {"bytes_written":3524,"cfile_cache_hit":11,"cfile_cache_hit_bytes":48052,"cfile_cache_miss":107,"cfile_cache_miss_bytes":2675440,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":1216,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":877,"drs_written":1,"lbm_read_time_us":4031,"lbm_reads_lt_1ms":135,"lbm_write_time_us":5254,"lbm_writes_lt_1ms":96,"num_input_rowsets":2,"peak_mem_usage":1324827,"reinsert_count":0,"rows_written":40128,"thread_start_us":75,"threads_started":1,"update_count":517}
I20240502 07:58:33.306048 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 748858 bytes on disk
I20240502 07:58:33.320350 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 07:58:33.321054 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.005529
I20240502 07:58:33.484529 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.433s	user 1.014s	sys 0.004s Metrics: {"bytes_written":3313,"cfile_cache_hit":14,"cfile_cache_hit_bytes":408280,"cfile_cache_miss":108,"cfile_cache_miss_bytes":2310258,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1900,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1633,"drs_written":1,"lbm_read_time_us":3694,"lbm_reads_lt_1ms":144,"lbm_write_time_us":6073,"lbm_writes_lt_1ms":96,"num_input_rowsets":2,"peak_mem_usage":1324225,"reinsert_count":0,"rows_written":40095,"thread_start_us":562,"threads_started":7,"update_count":484}
I20240502 07:58:33.485744 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 748549 bytes on disk
I20240502 07:58:33.486482 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 07:58:33.487124 30392 maintenance_manager.cc:419] P 133f2399e9b8473fbff0dfcab9a3d022: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.005206
I20240502 07:58:33.536643 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.215s	user 0.068s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5429,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":4342,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":677,"lbm_read_time_us":277,"lbm_reads_lt_1ms":20,"lbm_write_time_us":233,"lbm_writes_lt_1ms":8,"mutex_wait_us":4265,"reinsert_count":0,"update_count":517}
I20240502 07:58:33.659467 30282 maintenance_manager.cc:643] P 133f2399e9b8473fbff0dfcab9a3d022: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.172s	user 0.076s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5105,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":39,"lbm_read_time_us":404,"lbm_reads_lt_1ms":20,"lbm_write_time_us":323,"lbm_writes_lt_1ms":8,"mutex_wait_us":158,"reinsert_count":0,"update_count":484}
I20240502 07:58:33.930183 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:33.968348 30597 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:33.969650 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 07:58:33.972466 30597 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7fc84c635dcd  google::LogMessage::Fail() at ??:0
    @     0x7fc84c639b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fc84c6357cc  google::LogMessage::Flush() at ??:0
    @     0x7fc84c636f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fc84da175b1  kudu::operator-() at ??:0
    @     0x7fc84fafafd2  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7fc84fae405b  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7fc84fae3f90  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7fc84faba9bd  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7fc8506e6861  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7fc8506e593a  _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
    @     0x7fc8506ee149  _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
    @     0x7fc84b5b0d09  std::function<>::operator()() at ??:0
    @     0x7fc84b5ab107  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7fc84b5aaa38  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7fc84b5aa123  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7fc84b5d2b89  handle_request at ??:0
    @     0x7fc84b5d526b  worker_thread at ??:0
    @     0x7fc84e2b46db  start_thread at ??:0
    @     0x7fc84b7ec71f  clone at ??:0
W20240502 07:58:34.212622 30528 meta_cache.cc:302] tablet ddd2cd669a2b4b06a268464e4769d8db: replica 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503) has failed: Network error: recv got EOF from 127.29.83.66:39503 (error 108)
W20240502 07:58:34.223805 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.243691 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.257042 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.264021 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.286933 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.300623 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.326346 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.340200 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.372169 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.388553 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.427335 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.448124 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.494730 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.518461 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
I20240502 07:58:34.537611 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.271s	user 1.081s	sys 0.004s Metrics: {"bytes_written":735927,"cfile_cache_hit":13,"cfile_cache_hit_bytes":277026,"cfile_cache_miss":105,"cfile_cache_miss_bytes":2452320,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":832,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":919,"drs_written":1,"lbm_read_time_us":3896,"lbm_reads_lt_1ms":133,"lbm_write_time_us":6495,"lbm_writes_lt_1ms":92,"num_input_rowsets":2,"peak_mem_usage":1326345,"rows_written":40226,"spinlock_wait_cycles":167040,"thread_start_us":416,"threads_started":7}
I20240502 07:58:34.538561 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 755289 bytes on disk
I20240502 07:58:34.539032 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240502 07:58:34.551873 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.555809 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.558362 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.558755 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.560215 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.564832 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.570351 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.574378 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.575546 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.579066 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.588196 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.589398 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.598527 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.599779 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.610818 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.611785 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.638557 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.644340 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.650880 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.657814 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.660204 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.660204 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.682502 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.691073 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.693770 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.699131 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.715760 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.738173 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.753616 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.754158 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.758620 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.773731 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.785684 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.800882 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.820226 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.820631 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.823983 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.856616 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.874464 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.892222 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.892563 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.898725 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.956749 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.969149 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.974464 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.975427 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:34.984841 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.005482 30435 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.005482 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.006567 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.049408 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.078832 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.085409 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.129292 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.145576 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.169080 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.180058 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.209609 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.219245 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.219558 30528 meta_cache.cc:302] tablet ddd2cd669a2b4b06a268464e4769d8db: replica 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503) has failed: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
W20240502 07:58:35.258564 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.262166 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.280710 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.283797 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.296495 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.359441 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.375636 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.409197 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.414660 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.444777 30174 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50208: Illegal state: replica abeecd532ea6417fbb8200fd2c782145 is not leader of this config: current role FOLLOWER
I20240502 07:58:35.466637 30845 raft_consensus.cc:482] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.466768 30845 raft_consensus.cc:504] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.467784 30845 leader_election.cc:288] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
W20240502 07:58:35.468618 30150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
W20240502 07:58:35.469517 30150 leader_election.cc:334] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.472951 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1307 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:35.473109 30474 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 1.
I20240502 07:58:35.473464 30148 leader_election.cc:302] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 133f2399e9b8473fbff0dfcab9a3d022
I20240502 07:58:35.473697 30845 raft_consensus.cc:2781] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:35.473781 30845 raft_consensus.cc:482] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.473848 30845 raft_consensus.cc:3037] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:35.475178 30845 raft_consensus.cc:504] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.475692 30845 leader_election.cc:288] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Requested vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:35.476125 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2cd669a2b4b06a268464e4769d8db" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1307 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:35.476269 30474 raft_consensus.cc:3037] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:35.476648 30150 leader_election.cc:334] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.477563 30474 raft_consensus.cc:2454] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 2.
I20240502 07:58:35.477880 30148 leader_election.cc:302] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 133f2399e9b8473fbff0dfcab9a3d022
I20240502 07:58:35.478039 30845 raft_consensus.cc:2781] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:35.478235 30845 raft_consensus.cc:686] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [term 2 LEADER]: Becoming Leader. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Running, Role: LEADER
I20240502 07:58:35.478549 30845 consensus_queue.cc:237] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1307, Committed index: 1307, Last appended: 1.1307, Last appended by leader: 1307, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.480039 30845 raft_consensus.cc:482] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.480145 30845 raft_consensus.cc:504] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.480458 30845 leader_election.cc:288] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:35.480751 30845 raft_consensus.cc:482] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.481045 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1280 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:35.481194 30474 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 1.
I20240502 07:58:35.481351 30069 catalog_manager.cc:5484] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 reported cstate change: term changed from 1 to 2, leader changed from 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66) to abeecd532ea6417fbb8200fd2c782145 (127.29.83.65). New cstate: current_term: 2 leader_uuid: "abeecd532ea6417fbb8200fd2c782145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.481504 30852 raft_consensus.cc:482] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.481678 30148 leader_election.cc:302] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:35.481846 30850 raft_consensus.cc:2781] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:35.481936 30850 raft_consensus.cc:482] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.482004 30850 raft_consensus.cc:3037] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:35.482142 30150 leader_election.cc:334] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.482198 30845 raft_consensus.cc:504] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.482479 30845 leader_election.cc:288] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:35.482944 30854 raft_consensus.cc:482] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
I20240502 07:58:35.483026 30854 raft_consensus.cc:504] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.483098 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1283 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" is_pre_election: true
I20240502 07:58:35.483449 30474 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 1.
I20240502 07:58:35.483842 30148 leader_election.cc:302] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:35.484007 30845 raft_consensus.cc:2781] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:35.484063 30845 raft_consensus.cc:482] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 133f2399e9b8473fbff0dfcab9a3d022)
W20240502 07:58:35.484071 30150 leader_election.cc:334] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.484122 30845 raft_consensus.cc:3037] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:35.484071 30850 raft_consensus.cc:504] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.484517 30850 leader_election.cc:288] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Requested vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:35.481758 30852 raft_consensus.cc:504] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.485087 30845 raft_consensus.cc:504] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.485162 30474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1280 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:35.485281 30474 raft_consensus.cc:3037] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:35.485432 30845 leader_election.cc:288] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Requested vote from peers 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503), 6cfdd8e06ad74aa48ae8115d0ad2f473 (127.29.83.67:38759)
I20240502 07:58:35.485988 30854 leader_election.cc:288] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503)
I20240502 07:58:35.486299 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "abeecd532ea6417fbb8200fd2c782145" candidate_term: 2 candidate_status { last_received { term: 1 index: 1283 } } ignore_live_leader: false dest_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473"
I20240502 07:58:35.486393 30476 raft_consensus.cc:3037] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:35.486511 30474 raft_consensus.cc:2454] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 2.
W20240502 07:58:35.486626 30150 leader_election.cc:334] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.487016 30148 leader_election.cc:302] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 133f2399e9b8473fbff0dfcab9a3d022
I20240502 07:58:35.487080 30852 leader_election.cc:288] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abeecd532ea6417fbb8200fd2c782145 (127.29.83.65:40393), 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503)
I20240502 07:58:35.487397 30476 raft_consensus.cc:2454] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abeecd532ea6417fbb8200fd2c782145 in term 2.
I20240502 07:58:35.487571 30845 raft_consensus.cc:2781] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:35.487668 30845 raft_consensus.cc:686] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [term 2 LEADER]: Becoming Leader. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Running, Role: LEADER
I20240502 07:58:35.487823 30148 leader_election.cc:302] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473, abeecd532ea6417fbb8200fd2c782145; no voters: 
I20240502 07:58:35.487807 30845 consensus_queue.cc:237] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1280, Committed index: 1280, Last appended: 1.1280, Last appended by leader: 1280, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.488008 30850 raft_consensus.cc:2781] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:35.488212 30850 raft_consensus.cc:686] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [term 2 LEADER]: Becoming Leader. State: Replica: abeecd532ea6417fbb8200fd2c782145, State: Running, Role: LEADER
W20240502 07:58:35.488408 30150 leader_election.cc:334] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.488387 30850 consensus_queue.cc:237] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1283, Committed index: 1283, Last appended: 1.1283, Last appended by leader: 1283, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } }
I20240502 07:58:35.492720 30069 catalog_manager.cc:5484] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 reported cstate change: term changed from 1 to 2, leader changed from 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66) to abeecd532ea6417fbb8200fd2c782145 (127.29.83.65). New cstate: current_term: 2 leader_uuid: "abeecd532ea6417fbb8200fd2c782145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.493090 30069 catalog_manager.cc:5484] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 reported cstate change: term changed from 1 to 2, leader changed from 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66) to abeecd532ea6417fbb8200fd2c782145 (127.29.83.65). New cstate: current_term: 2 leader_uuid: "abeecd532ea6417fbb8200fd2c782145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abeecd532ea6417fbb8200fd2c782145" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 40393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "133f2399e9b8473fbff0dfcab9a3d022" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:35.495277 30412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.496044 30211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096d782a2b994f25a0269a0e6f15bbbd" candidate_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" candidate_term: 2 candidate_status { last_received { term: 1 index: 1280 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145" is_pre_election: true
I20240502 07:58:35.496044 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128a446093e425a8fa63ef6d0ed7224" candidate_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" candidate_term: 2 candidate_status { last_received { term: 1 index: 1283 } } ignore_live_leader: false dest_uuid: "abeecd532ea6417fbb8200fd2c782145" is_pre_election: true
W20240502 07:58:35.496894 30412 leader_election.cc:334] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.497000 30412 leader_election.cc:302] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473; no voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145
I20240502 07:58:35.497789 30852 raft_consensus.cc:2726] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 07:58:35.497875 30412 leader_election.cc:334] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111)
I20240502 07:58:35.497947 30412 leader_election.cc:302] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cfdd8e06ad74aa48ae8115d0ad2f473; no voters: 133f2399e9b8473fbff0dfcab9a3d022, abeecd532ea6417fbb8200fd2c782145
I20240502 07:58:35.498067 30852 raft_consensus.cc:2726] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:58:35.504861 30476 raft_consensus.cc:1261] T ddd2cd669a2b4b06a268464e4769d8db P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Refusing update from remote peer abeecd532ea6417fbb8200fd2c782145: Log matching property violated. Preceding OpId in replica: term: 1 index: 1307. Preceding OpId from leader: term: 2 index: 1309. (index mismatch)
I20240502 07:58:35.505350 30858 consensus_queue.cc:1035] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1308, Last known committed idx: 1307, Time since last communication: 0.000s
W20240502 07:58:35.505645 30150 consensus_peers.cc:485] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.525630 30476 raft_consensus.cc:1261] T 096d782a2b994f25a0269a0e6f15bbbd P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Refusing update from remote peer abeecd532ea6417fbb8200fd2c782145: Log matching property violated. Preceding OpId in replica: term: 1 index: 1280. Preceding OpId from leader: term: 2 index: 1282. (index mismatch)
I20240502 07:58:35.525851 30474 raft_consensus.cc:1261] T 4128a446093e425a8fa63ef6d0ed7224 P 6cfdd8e06ad74aa48ae8115d0ad2f473 [term 2 FOLLOWER]: Refusing update from remote peer abeecd532ea6417fbb8200fd2c782145: Log matching property violated. Preceding OpId in replica: term: 1 index: 1283. Preceding OpId from leader: term: 2 index: 1285. (index mismatch)
I20240502 07:58:35.526602 30845 consensus_queue.cc:1035] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1281, Last known committed idx: 1280, Time since last communication: 0.000s
I20240502 07:58:35.526770 30858 consensus_queue.cc:1035] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfdd8e06ad74aa48ae8115d0ad2f473" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1284, Last known committed idx: 1283, Time since last communication: 0.000s
W20240502 07:58:35.526881 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:35.526938 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.702379 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:35.731410 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:35.869422 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
W20240502 07:58:35.873639 30436 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50762: Illegal state: replica 6cfdd8e06ad74aa48ae8115d0ad2f473 is not leader of this config: current role FOLLOWER
I20240502 07:58:37.478542 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:37.499213 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:37.966109 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 07:58:38.045826 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
I20240502 07:58:38.067301 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.000000
W20240502 07:58:38.102939 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:38.313561 30150 consensus_peers.cc:485] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 07:58:38.314551 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.268s	user 0.238s	sys 0.004s Metrics: {"bytes_written":149370,"cfile_init":1,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1488,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8190,"thread_start_us":86,"threads_started":1}
I20240502 07:58:38.315575 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014457
I20240502 07:58:38.341501 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: FlushMRSOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.274s	user 0.222s	sys 0.004s Metrics: {"bytes_written":151442,"cfile_init":1,"dirs.queue_time_us":1171,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1975,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1618,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8316,"thread_start_us":87,"threads_started":1}
I20240502 07:58:38.342423 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=1.014457
I20240502 07:58:39.627060 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023152003471450112 Inserted count: 75700
I20240502 07:58:39.627139 30029 linked_list-test-util.h:421] Finished inserting list. Added 75700 in chain
I20240502 07:58:39.627156 30029 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 07:58:39.627171 30029 linked_list-test-util.h:424] 0: -5931160805462835200
I20240502 07:58:39.627185 30029 linked_list-test-util.h:424] 1: -228811295755665407
I20240502 07:58:39.627199 30029 linked_list-test-util.h:424] 2: 5473960426416504834
I20240502 07:58:39.627215 30029 linked_list-test-util.h:424] 3: -7270152662609166333
I20240502 07:58:39.627229 30029 linked_list-test-util.h:424] 4: -7498682483388121084
I20240502 07:58:39.627245 30029 linked_list-test-util.h:424] 5: -1796332973680951291
I20240502 07:58:39.627260 30029 linked_list-test-util.h:424] 6: 3906438748491218950
I20240502 07:58:39.627274 30029 linked_list-test-util.h:424] 7: -8837674340534452217
I20240502 07:58:39.627288 30029 linked_list-test-util.h:424] 8: -9066204161313406968
I20240502 07:58:39.627301 30029 linked_list-test-util.h:424] 9: -3363854651606237175
I20240502 07:58:39.627316 30029 linked_list-test-util.h:424] 10: 2338776333077643274
I20240502 07:58:39.627331 30029 linked_list-test-util.h:424] 11: 8041548055249813515
I20240502 07:58:39.627346 30029 linked_list-test-util.h:424] 12: 7813018234470858764
I20240502 07:58:39.627359 30029 linked_list-test-util.h:424] 13: -4931376329531523059
I20240502 07:58:39.627374 30029 linked_list-test-util.h:424] 14: 771254655152357390
I20240502 07:58:39.627389 30029 linked_list-test-util.h:424] 15: 6474026377324527631
I20240502 07:58:39.627418 30029 linked_list-test-util.h:424] 16: 6245215081568862224
I20240502 07:58:39.627434 30029 linked_list-test-util.h:424] 17: -6498898007456808943
I20240502 07:58:39.627449 30029 linked_list-test-util.h:424] 18: -796267022772928494
I20240502 07:58:39.627465 30029 linked_list-test-util.h:424] 19: 4906223224422531091
I20240502 07:58:39.627480 30029 linked_list-test-util.h:424] 20: 4677552666155286548
I20240502 07:58:39.627494 30029 linked_list-test-util.h:424] 21: -8066419685382094827
I20240502 07:58:39.627509 30029 linked_list-test-util.h:424] 22: -2363788700698214378
I20240502 07:58:39.627524 30029 linked_list-test-util.h:424] 23: 3338701546497245207
I20240502 07:58:39.627540 30029 linked_list-test-util.h:424] 24: 3110030988230000664
I20240502 07:58:39.627555 30029 linked_list-test-util.h:424] 25: 8812802710402170905
I20240502 07:58:39.627571 30029 linked_list-test-util.h:424] 26: -3931310378623500262
I20240502 07:58:39.627586 30029 linked_list-test-util.h:424] 27: -4159840199402455013
I20240502 07:58:39.627601 30029 linked_list-test-util.h:424] 28: 1542509310304714780
I20240502 07:58:39.627617 30029 linked_list-test-util.h:424] 29: 7245281032476885021
I20240502 07:58:39.627632 30029 linked_list-test-util.h:424] 30: -5498832056548786146
I20240502 07:58:39.627646 30029 linked_list-test-util.h:424] 31: -5727643352304451553
I20240502 07:58:39.627662 30029 linked_list-test-util.h:424] 32: -25012367620571104
I20240502 07:58:39.627678 30029 linked_list-test-util.h:424] 33: 5677759354551599137
I20240502 07:58:39.627693 30029 linked_list-test-util.h:424] 34: -7066353734474072030
I20240502 07:58:39.627708 30029 linked_list-test-util.h:424] 35: -7295165030229737437
I20240502 07:58:39.627723 30029 linked_list-test-util.h:424] 36: -1592534045545856988
I20240502 07:58:39.627738 30029 linked_list-test-util.h:424] 37: 4110237676626313253
I20240502 07:58:39.627753 30029 linked_list-test-util.h:424] 38: -8634156887376068570
I20240502 07:58:39.627769 30029 linked_list-test-util.h:424] 39: -8862686708155023321
I20240502 07:58:39.627784 30029 linked_list-test-util.h:424] 40: -3160055723471142872
I20240502 07:58:39.627799 30029 linked_list-test-util.h:424] 41: 2542575261212737577
I20240502 07:58:39.627815 30029 linked_list-test-util.h:424] 42: 8245065508408197162
I20240502 07:58:39.627830 30029 linked_list-test-util.h:424] 43: 8016535687629242411
I20240502 07:58:39.627844 30029 linked_list-test-util.h:424] 44: -4727577401396428756
I20240502 07:58:39.627864 30029 linked_list-test-util.h:424] 45: 975053583287451693
I20240502 07:58:39.627879 30029 linked_list-test-util.h:424] 46: 6677543830482911278
I20240502 07:58:39.627894 30029 linked_list-test-util.h:424] 47: 6448873272215666735
I20240502 07:58:39.628559 30029 linked_list-test-util.h:424] 48: -6295099079321714640
I20240502 07:58:39.628612 30029 linked_list-test-util.h:424] 49: -592468094637834191
I20240502 07:58:39.713855 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.398s	user 1.320s	sys 0.004s Metrics: {"bytes_written":8369,"cfile_cache_hit":15,"cfile_cache_hit_bytes":498809,"cfile_cache_miss":125,"cfile_cache_miss_bytes":2811539,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":11299,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":979,"drs_written":1,"lbm_read_time_us":4751,"lbm_reads_lt_1ms":157,"lbm_write_time_us":7475,"lbm_writes_lt_1ms":109,"num_input_rowsets":2,"peak_mem_usage":1588664,"reinsert_count":0,"rows_written":48416,"thread_start_us":400,"threads_started":7,"update_count":1286}
I20240502 07:58:39.715516 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 904029 bytes on disk
I20240502 07:58:39.716183 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 07:58:39.716764 30523 maintenance_manager.cc:419] P 6cfdd8e06ad74aa48ae8115d0ad2f473: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.010861
I20240502 07:58:39.809744 30413 maintenance_manager.cc:643] P 6cfdd8e06ad74aa48ae8115d0ad2f473: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.093s	user 0.087s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":12351,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":49,"lbm_read_time_us":261,"lbm_reads_lt_1ms":20,"lbm_write_time_us":267,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1286}
I20240502 07:58:39.811228 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636699609051 usec, L: 1 Expected Rows: 28000
I20240502 07:58:39.824970 30029 linked_list-test-util.h:528] Calling callback on tserver 6cfdd8e06ad74aa48ae8115d0ad2f473
I20240502 07:58:39.927470 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:39.960229 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:39.991585 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: CompactRowSetsOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 1.648s	user 1.160s	sys 0.004s Metrics: {"bytes_written":8549,"cfile_cache_hit":16,"cfile_cache_hit_bytes":498123,"cfile_cache_miss":128,"cfile_cache_miss_bytes":2811313,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":10628,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":7958,"drs_written":1,"lbm_read_time_us":7841,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":163,"lbm_write_time_us":6900,"lbm_writes_lt_1ms":109,"mutex_wait_us":271,"num_input_rowsets":2,"peak_mem_usage":1589041,"reinsert_count":0,"rows_written":48444,"thread_start_us":633,"threads_started":7,"update_count":1314}
I20240502 07:58:39.992440 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db): 904582 bytes on disk
I20240502 07:58:39.992820 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: UndoDeltaBlockGCOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 07:58:39.993294 30261 maintenance_manager.cc:419] P abeecd532ea6417fbb8200fd2c782145: Scheduling MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db): perf score=0.011088
I20240502 07:58:40.104282 30151 maintenance_manager.cc:643] P abeecd532ea6417fbb8200fd2c782145: MajorDeltaCompactionOp(ddd2cd669a2b4b06a268464e4769d8db) complete. Timing: real 0.111s	user 0.103s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":12580,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":196,"lbm_read_time_us":304,"lbm_reads_lt_1ms":20,"lbm_write_time_us":280,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1314}
W20240502 07:58:40.183404 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 07:58:40.442564 30029 linked_list-test-util.h:767] Done collecting results (28000 rows in 617.691ms)
I20240502 07:58:40.462167 30029 linked_list-test-util.h:556] Snapshot: P: 1714636699609051 usec, L: 1 verified. Result: OK
I20240502 07:58:40.464426 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636709604131 usec, L: 1 Expected Rows: 53700
I20240502 07:58:40.469962 30029 linked_list-test-util.h:528] Calling callback on tserver 6cfdd8e06ad74aa48ae8115d0ad2f473
W20240502 07:58:40.524572 30150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240502 07:58:40.525296 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:40.857656 30150 consensus_peers.cc:485] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 07:58:41.222503 30029 linked_list-test-util.h:767] Done collecting results (53700 rows in 752.612ms)
I20240502 07:58:41.261466 30029 linked_list-test-util.h:556] Snapshot: P: 1714636709604131 usec, L: 1 verified. Result: OK
I20240502 07:58:41.264015 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636719597522 usec, L: 1 Expected Rows: 75700
I20240502 07:58:41.268669 30029 linked_list-test-util.h:528] Calling callback on tserver 6cfdd8e06ad74aa48ae8115d0ad2f473
I20240502 07:58:41.585693 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:41.606472 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:42.411185 30029 linked_list-test-util.h:767] Done collecting results (75700 rows in 1142.6ms)
I20240502 07:58:42.503837 30029 linked_list-test-util.h:556] Snapshot: P: 1714636719597522 usec, L: 1 verified. Result: OK
I20240502 07:58:42.506486 30029 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 75700
W20240502 07:58:42.862172 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 07:58:43.070439 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:43.079356 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 07:58:43.094867 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:43.293496 30150 consensus_peers.cc:485] T ddd2cd669a2b4b06a268464e4769d8db P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 07:58:44.114253 30029 linked_list-test-util.h:767] Done collecting results (75700 rows in 1228.27ms)
I20240502 07:58:44.169795 30029 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240502 07:58:44.170123 30029 linked_list-test-util.h:692] Successfully verified 75700 rows
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: 133f2399e9b8473fbff0dfcab9a3d022, Rpc address: host: "127.29.83.66" port: 39503 (Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111))
I20240502 07:58:44.187458 30029 webui_checker.cc:88] shutting down CURL thread
I20240502 07:58:44.506026 30561 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 07:58:44.525735 30596 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 07:58:45.271617 30150 consensus_peers.cc:485] T 096d782a2b994f25a0269a0e6f15bbbd P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:45.328649 30150 consensus_peers.cc:485] T 4128a446093e425a8fa63ef6d0ed7224 P abeecd532ea6417fbb8200fd2c782145 -> Peer 133f2399e9b8473fbff0dfcab9a3d022 (127.29.83.66:39503): Couldn't send request to peer 133f2399e9b8473fbff0dfcab9a3d022. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:45.407672 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30132
I20240502 07:58:45.419142 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30394
I20240502 07:58:45.427450 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30041
2024-05-02T07:58:45Z chronyd exiting
I20240502 07:58:45.445573 30029 test_util.cc:175] -----------------------------------------------
I20240502 07:58:45.445643 30029 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714636688565791-30029-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (36870 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240502 07:58:45.446254 30029 test_util.cc:268] Using random seed: -118462870
I20240502 07:58:45.446926 30029 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 07:58:45.446970 30029 linked_list-test.cc:101] --------------
I20240502 07:58:45.446988 30029 linked_list-test.cc:102] 50 chains
I20240502 07:58:45.447000 30029 linked_list-test.cc:103] 3 tablets
I20240502 07:58:45.447010 30029 linked_list-test.cc:104] Mutations on
I20240502 07:58:45.447021 30029 linked_list-test.cc:105] --------------
I20240502 07:58:45.447065 30029 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:45.447091 30029 ts_itest-base.cc:113] --------------
I20240502 07:58:45.447104 30029 ts_itest-base.cc:114] 3 tablet servers
I20240502 07:58:45.447118 30029 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:45.447134 30029 ts_itest-base.cc:116] --------------
2024-05-02T07:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:45Z Disabled control of system clock
I20240502 07:58:45.455595 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.126:33021
--webserver_interface=127.29.83.126
--webserver_port=0
--builtin_ntp_servers=127.29.83.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.126:33021
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 07:58:45.572089 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:45.572342 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:45.572397 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:45.575578 30919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:45.575649 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:45.575670 30919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:45.575687 30919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:45.579538 30919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.126:33021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.126:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.29.83.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.30919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:45.580621 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:45.581853 30919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:45.590936 30927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:45.591715 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:45.592455 30925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:45.593271 30919 server_base.cc:1034] running on GCE node
I20240502 07:58:45.593825 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:45.594861 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:45.596015 30919 hybrid_clock.cc:648] HybridClock initialized: now 1714636725595992 us; error 45 us; skew 500 ppm
I20240502 07:58:45.602183 30919 webserver.cc:460] Webserver started at http://127.29.83.126:37013/ using document root <none> and password file <none>
I20240502 07:58:45.602828 30919 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:45.602895 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:45.603123 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:45.605159 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/instance:
uuid: "52000ffdd7fb45b290a248682da109ab"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.605655 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal/instance:
uuid: "52000ffdd7fb45b290a248682da109ab"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.609200 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 07:58:45.611543 30933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:45.612695 30919 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:58:45.612823 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
uuid: "52000ffdd7fb45b290a248682da109ab"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.612918 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:45.633926 30919 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:45.634653 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:45.634811 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:45.641536 30919 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.126:33021
I20240502 07:58:45.641558 30985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.126:33021 every 8 connection(s)
I20240502 07:58:45.642489 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/data/info.pb
I20240502 07:58:45.645229 30986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:45.650585 30986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab: Bootstrap starting.
I20240502 07:58:45.652014 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 30919
I20240502 07:58:45.652153 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/master-0/wal/instance
I20240502 07:58:45.653247 30986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:45.654122 30986 log.cc:824] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:45.655761 30986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab: No bootstrap required, opened a new log
I20240502 07:58:45.658398 30986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } }
I20240502 07:58:45.658596 30986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:45.658666 30986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52000ffdd7fb45b290a248682da109ab, State: Initialized, Role: FOLLOWER
I20240502 07:58:45.659096 30986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } }
I20240502 07:58:45.659222 30986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:45.659288 30986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:45.659408 30986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:45.660185 30986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } }
I20240502 07:58:45.660501 30986 leader_election.cc:302] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52000ffdd7fb45b290a248682da109ab; no voters: 
I20240502 07:58:45.660784 30986 leader_election.cc:288] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 07:58:45.660873 30991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:45.661480 30991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [term 1 LEADER]: Becoming Leader. State: Replica: 52000ffdd7fb45b290a248682da109ab, State: Running, Role: LEADER
I20240502 07:58:45.661981 30991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } }
I20240502 07:58:45.662055 30986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:45.663656 30993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52000ffdd7fb45b290a248682da109ab. Latest consensus state: current_term: 1 leader_uuid: "52000ffdd7fb45b290a248682da109ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } } }
I20240502 07:58:45.663797 30993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:45.663657 30992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52000ffdd7fb45b290a248682da109ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52000ffdd7fb45b290a248682da109ab" member_type: VOTER last_known_addr { host: "127.29.83.126" port: 33021 } } }
I20240502 07:58:45.663865 30992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:45.664165 31005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:45.667054 31005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:45.672154 31005 catalog_manager.cc:1347] Generated new cluster ID: 1aeb32a14bcd4613807ef0ff64d51866
I20240502 07:58:45.672271 31005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:45.683669 31005 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:45.684659 31005 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:45.692852 31005 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 52000ffdd7fb45b290a248682da109ab: Generated new TSK 0
I20240502 07:58:45.693686 31005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:45.704525 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.65:0
--local_ip_for_outbound_sockets=127.29.83.65
--webserver_interface=127.29.83.65
--webserver_port=0
--tserver_master_addrs=127.29.83.126:33021
--builtin_ntp_servers=127.29.83.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 07:58:45.820626 31010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:45.820888 31010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:45.820945 31010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:45.824311 31010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:45.824441 31010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.65
I20240502 07:58:45.828367 31010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.29.83.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.29.83.126:33021
--never_fsync=true
--heap_profile_path=/tmp/kudu.31010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.65
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:45.829321 31010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:45.830478 31010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:45.839646 31016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:45.840502 31015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:45.841205 31018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:45.842119 31010 server_base.cc:1034] running on GCE node
I20240502 07:58:45.842608 31010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:45.843225 31010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:45.844420 31010 hybrid_clock.cc:648] HybridClock initialized: now 1714636725844404 us; error 32 us; skew 500 ppm
I20240502 07:58:45.846735 31010 webserver.cc:460] Webserver started at http://127.29.83.65:46637/ using document root <none> and password file <none>
I20240502 07:58:45.847612 31010 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:45.847683 31010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:45.847880 31010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:45.849886 31010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/instance:
uuid: "70e6a2ff133a488799712f7831863d51"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.850373 31010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/instance:
uuid: "70e6a2ff133a488799712f7831863d51"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.854030 31010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.002s
I20240502 07:58:45.856365 31024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:45.857466 31010 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:58:45.857589 31010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
uuid: "70e6a2ff133a488799712f7831863d51"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:58:45.857689 31010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:45.872701 31010 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:45.873325 31010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:45.873489 31010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:45.874023 31010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:45.874904 31010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:45.874951 31010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:45.875011 31010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:45.875036 31010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:45.883518 31010 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.65:39701
I20240502 07:58:45.883580 31137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.65:39701 every 8 connection(s)
I20240502 07:58:45.884490 31010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
I20240502 07:58:45.891263 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 31010
I20240502 07:58:45.891398 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/instance
I20240502 07:58:45.893023 31138 heartbeater.cc:347] Connected to a master server at 127.29.83.126:33021
I20240502 07:58:45.893257 31138 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:45.893915 31138 heartbeater.cc:511] Master 127.29.83.126:33021 requested a full tablet report, sending...
I20240502 07:58:45.893968 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.66:0
--local_ip_for_outbound_sockets=127.29.83.66
--webserver_interface=127.29.83.66
--webserver_port=0
--tserver_master_addrs=127.29.83.126:33021
--builtin_ntp_servers=127.29.83.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 07:58:45.895458 30950 ts_manager.cc:194] Registered new tserver with Master: 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701)
I20240502 07:58:45.896741 30950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.65:34459
W20240502 07:58:46.019184 31141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:46.019440 31141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:46.019522 31141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:46.022917 31141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:46.023118 31141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.66
I20240502 07:58:46.027158 31141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.29.83.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.29.83.126:33021
--never_fsync=true
--heap_profile_path=/tmp/kudu.31141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.66
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:46.028345 31141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:46.029631 31141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:46.038887 31146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:46.039405 31147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:46.039986 31149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:46.040661 31141 server_base.cc:1034] running on GCE node
I20240502 07:58:46.041255 31141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:46.041934 31141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:46.043108 31141 hybrid_clock.cc:648] HybridClock initialized: now 1714636726043093 us; error 31 us; skew 500 ppm
I20240502 07:58:46.045215 31141 webserver.cc:460] Webserver started at http://127.29.83.66:34571/ using document root <none> and password file <none>
I20240502 07:58:46.045819 31141 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:46.045876 31141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:46.046061 31141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:46.047853 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/instance:
uuid: "d64238ec658a4dddaee6986e184db7d1"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.048398 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal/instance:
uuid: "d64238ec658a4dddaee6986e184db7d1"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.051887 31141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 07:58:46.054275 31155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.055569 31141 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240502 07:58:46.055680 31141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
uuid: "d64238ec658a4dddaee6986e184db7d1"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.055804 31141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:46.090543 31141 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:46.091198 31141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:46.091356 31141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:46.091948 31141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:46.092831 31141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:46.092876 31141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.092919 31141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:46.092952 31141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.101333 31141 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.66:39375
I20240502 07:58:46.101362 31268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.66:39375 every 8 connection(s)
I20240502 07:58:46.102262 31141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/data/info.pb
I20240502 07:58:46.110605 31269 heartbeater.cc:347] Connected to a master server at 127.29.83.126:33021
I20240502 07:58:46.110844 31269 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:46.111416 31269 heartbeater.cc:511] Master 127.29.83.126:33021 requested a full tablet report, sending...
I20240502 07:58:46.112241 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 31141
I20240502 07:58:46.112358 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-1/wal/instance
I20240502 07:58:46.112432 30950 ts_manager.cc:194] Registered new tserver with Master: d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375)
I20240502 07:58:46.112987 30950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.66:40295
I20240502 07:58:46.114215 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.67:0
--local_ip_for_outbound_sockets=127.29.83.67
--webserver_interface=127.29.83.67
--webserver_port=0
--tserver_master_addrs=127.29.83.126:33021
--builtin_ntp_servers=127.29.83.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 07:58:46.222594 31272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:46.222831 31272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:46.222882 31272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:46.226222 31272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:46.226348 31272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.67
I20240502 07:58:46.230171 31272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.29.83.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.29.83.126:33021
--never_fsync=true
--heap_profile_path=/tmp/kudu.31272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.67
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:46.231134 31272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:46.232319 31272 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:46.242165 31278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:46.242738 31277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:46.243335 31280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:46.244076 31272 server_base.cc:1034] running on GCE node
I20240502 07:58:46.244560 31272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:46.245185 31272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:46.246410 31272 hybrid_clock.cc:648] HybridClock initialized: now 1714636726246325 us; error 102 us; skew 500 ppm
I20240502 07:58:46.248836 31272 webserver.cc:460] Webserver started at http://127.29.83.67:33353/ using document root <none> and password file <none>
I20240502 07:58:46.249444 31272 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:46.249512 31272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:46.249711 31272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:46.251433 31272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/instance:
uuid: "c466253287d7465e80c44f03eba6ceb8"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.251991 31272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal/instance:
uuid: "c466253287d7465e80c44f03eba6ceb8"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.255494 31272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 07:58:46.257941 31286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.259092 31272 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240502 07:58:46.259222 31272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
uuid: "c466253287d7465e80c44f03eba6ceb8"
format_stamp: "Formatted at 2024-05-02 07:58:46 on dist-test-slave-2kd1"
I20240502 07:58:46.259332 31272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:46.282343 31272 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:58:46.282987 31272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:46.283130 31272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:46.283843 31272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:46.285652 31272 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:46.285753 31272 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.285831 31272 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:46.285856 31272 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:58:46.296456 31272 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.67:43779
I20240502 07:58:46.296453 31399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.67:43779 every 8 connection(s)
I20240502 07:58:46.297406 31272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/data/info.pb
I20240502 07:58:46.300354 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 31272
I20240502 07:58:46.300495 30029 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-2/wal/instance
I20240502 07:58:46.306264 31400 heartbeater.cc:347] Connected to a master server at 127.29.83.126:33021
I20240502 07:58:46.306483 31400 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:46.307000 31400 heartbeater.cc:511] Master 127.29.83.126:33021 requested a full tablet report, sending...
I20240502 07:58:46.307986 30949 ts_manager.cc:194] Registered new tserver with Master: c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779)
I20240502 07:58:46.308542 30949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.67:33513
I20240502 07:58:46.314515 30029 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 07:58:46.323264 30949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41942:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240502 07:58:46.325898 30949 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:58:46.341693 31072 tablet_service.cc:1467] Processing CreateTablet for tablet e127013489864087b78e4e2f68290954 (DEFAULT_TABLE table=TestTable [id=33131ce882ed48daa0eba3936222d661]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 07:58:46.341704 31203 tablet_service.cc:1467] Processing CreateTablet for tablet e127013489864087b78e4e2f68290954 (DEFAULT_TABLE table=TestTable [id=33131ce882ed48daa0eba3936222d661]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 07:58:46.341696 31334 tablet_service.cc:1467] Processing CreateTablet for tablet e127013489864087b78e4e2f68290954 (DEFAULT_TABLE table=TestTable [id=33131ce882ed48daa0eba3936222d661]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 07:58:46.342583 31072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e127013489864087b78e4e2f68290954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:46.342592 31203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e127013489864087b78e4e2f68290954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:46.342590 31334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e127013489864087b78e4e2f68290954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:46.348340 31417 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Bootstrap starting.
I20240502 07:58:46.349398 31418 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Bootstrap starting.
I20240502 07:58:46.349820 31419 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Bootstrap starting.
I20240502 07:58:46.350800 31417 tablet_bootstrap.cc:654] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:46.351291 31418 tablet_bootstrap.cc:654] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:46.351611 31417 log.cc:824] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:46.351876 31419 tablet_bootstrap.cc:654] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:46.351903 31418 log.cc:824] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:46.352592 31419 log.cc:824] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:46.353467 31418 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: No bootstrap required, opened a new log
I20240502 07:58:46.353468 31417 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: No bootstrap required, opened a new log
I20240502 07:58:46.353648 31417 ts_tablet_manager.cc:1367] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 07:58:46.353648 31418 ts_tablet_manager.cc:1367] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 07:58:46.354053 31419 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: No bootstrap required, opened a new log
I20240502 07:58:46.354209 31419 ts_tablet_manager.cc:1367] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 07:58:46.357321 31419 raft_consensus.cc:348] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.357522 31419 raft_consensus.cc:374] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:46.357601 31419 raft_consensus.cc:729] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64238ec658a4dddaee6986e184db7d1, State: Initialized, Role: FOLLOWER
I20240502 07:58:46.357534 31417 raft_consensus.cc:348] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.357534 31418 raft_consensus.cc:348] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.357736 31417 raft_consensus.cc:374] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:46.357745 31418 raft_consensus.cc:374] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:46.357805 31418 raft_consensus.cc:729] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c466253287d7465e80c44f03eba6ceb8, State: Initialized, Role: FOLLOWER
I20240502 07:58:46.357853 31417 raft_consensus.cc:729] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e6a2ff133a488799712f7831863d51, State: Initialized, Role: FOLLOWER
I20240502 07:58:46.358107 31419 consensus_queue.cc:260] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.358250 31417 consensus_queue.cc:260] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.358237 31418 consensus_queue.cc:260] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.358922 31400 heartbeater.cc:503] Master 127.29.83.126:33021 was elected leader, sending a full tablet report...
I20240502 07:58:46.359042 31417 ts_tablet_manager.cc:1398] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20240502 07:58:46.359042 31418 ts_tablet_manager.cc:1398] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.006s
I20240502 07:58:46.359048 31419 ts_tablet_manager.cc:1398] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
I20240502 07:58:46.359362 31138 heartbeater.cc:503] Master 127.29.83.126:33021 was elected leader, sending a full tablet report...
I20240502 07:58:46.359365 31269 heartbeater.cc:503] Master 127.29.83.126:33021 was elected leader, sending a full tablet report...
W20240502 07:58:46.385608 31139 tablet.cc:2369] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:46.548333 31401 tablet.cc:2369] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:46.552254 31423 raft_consensus.cc:482] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:46.552485 31423 raft_consensus.cc:504] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.553831 31423 leader_election.cc:288] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779), 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701)
I20240502 07:58:46.557621 31354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e127013489864087b78e4e2f68290954" candidate_uuid: "d64238ec658a4dddaee6986e184db7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c466253287d7465e80c44f03eba6ceb8" is_pre_election: true
I20240502 07:58:46.557617 31092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e127013489864087b78e4e2f68290954" candidate_uuid: "d64238ec658a4dddaee6986e184db7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e6a2ff133a488799712f7831863d51" is_pre_election: true
I20240502 07:58:46.557902 31092 raft_consensus.cc:2454] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d64238ec658a4dddaee6986e184db7d1 in term 0.
I20240502 07:58:46.557904 31354 raft_consensus.cc:2454] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d64238ec658a4dddaee6986e184db7d1 in term 0.
I20240502 07:58:46.558359 31159 leader_election.cc:302] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c466253287d7465e80c44f03eba6ceb8, d64238ec658a4dddaee6986e184db7d1; no voters: 
I20240502 07:58:46.558619 31423 raft_consensus.cc:2781] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:46.558697 31423 raft_consensus.cc:482] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:46.558754 31423 raft_consensus.cc:3037] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:46.559935 31423 raft_consensus.cc:504] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.560402 31423 leader_election.cc:288] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [CANDIDATE]: Term 1 election: Requested vote from peers c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779), 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701)
I20240502 07:58:46.560820 31354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e127013489864087b78e4e2f68290954" candidate_uuid: "d64238ec658a4dddaee6986e184db7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c466253287d7465e80c44f03eba6ceb8"
I20240502 07:58:46.560822 31092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e127013489864087b78e4e2f68290954" candidate_uuid: "d64238ec658a4dddaee6986e184db7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e6a2ff133a488799712f7831863d51"
I20240502 07:58:46.560940 31354 raft_consensus.cc:3037] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:46.560961 31092 raft_consensus.cc:3037] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:46.561931 31092 raft_consensus.cc:2454] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d64238ec658a4dddaee6986e184db7d1 in term 1.
I20240502 07:58:46.562000 31354 raft_consensus.cc:2454] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d64238ec658a4dddaee6986e184db7d1 in term 1.
I20240502 07:58:46.562322 31157 leader_election.cc:302] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e6a2ff133a488799712f7831863d51, d64238ec658a4dddaee6986e184db7d1; no voters: 
I20240502 07:58:46.562592 31423 raft_consensus.cc:2781] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:46.562860 31423 raft_consensus.cc:686] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [term 1 LEADER]: Becoming Leader. State: Replica: d64238ec658a4dddaee6986e184db7d1, State: Running, Role: LEADER
I20240502 07:58:46.563350 31423 consensus_queue.cc:237] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:58:46.566517 30950 catalog_manager.cc:5484] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d64238ec658a4dddaee6986e184db7d1 (127.29.83.66). New cstate: current_term: 1 leader_uuid: "d64238ec658a4dddaee6986e184db7d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:46.603359 31270 tablet.cc:2369] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:46.626220 30029 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 07:58:46.627785 30029 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 70e6a2ff133a488799712f7831863d51 to finish bootstrapping
I20240502 07:58:46.633594 30029 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d64238ec658a4dddaee6986e184db7d1 to finish bootstrapping
I20240502 07:58:46.639046 30029 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c466253287d7465e80c44f03eba6ceb8 to finish bootstrapping
I20240502 07:58:46.644285 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 31010
I20240502 07:58:46.654464 30029 linked_list-test-util.h:401] Running for: 30.000s
I20240502 07:58:46.654541 30029 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 07:58:46.654563 30029 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
W20240502 07:58:46.672359 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111)
I20240502 07:58:46.672751 31354 raft_consensus.cc:1261] T e127013489864087b78e4e2f68290954 P c466253287d7465e80c44f03eba6ceb8 [term 1 FOLLOWER]: Refusing update from remote peer d64238ec658a4dddaee6986e184db7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 07:58:46.673350 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:46.673486 31427 consensus_queue.cc:1035] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:46.678623 31432 mvcc.cc:205] Tried to move back new op lower bound from 7023152032443994112 to 7023152032006541312. Current Snapshot: MvccSnapshot[applied={T|T < 7023152032443994112}]
W20240502 07:58:48.954771 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 07:58:50.548395 31423 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 67 ops behind the committed index 
W20240502 07:58:51.652184 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:52.158481 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 07:58:54.031011 31427 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 540 ops behind the committed index  [suppressed 6 similar messages]
W20240502 07:58:54.036554 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:56.620601 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:56.655349 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023152073321721856 Inserted count: 69500
I20240502 07:58:57.126463 31478 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 966 ops behind the committed index  [suppressed 5 similar messages]
W20240502 07:58:57.703658 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 07:58:59.494561 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 07:59:00.419266 31478 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 1432 ops behind the committed index  [suppressed 5 similar messages]
W20240502 07:59:01.826817 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 07:59:02.786473 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 07:59:03.842229 31533 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 1922 ops behind the committed index  [suppressed 6 similar messages]
W20240502 07:59:04.275532 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 07:59:06.655458 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023152114284376064 Inserted count: 139050
W20240502 07:59:06.803769 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 07:59:07.196285 31437 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 2381 ops behind the committed index  [suppressed 6 similar messages]
W20240502 07:59:07.810227 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 07:59:09.201435 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 07:59:10.266391 31566 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 2829 ops behind the committed index  [suppressed 5 similar messages]
W20240502 07:59:11.924646 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:59:12.848731 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 07:59:13.351716 31558 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 3265 ops behind the committed index  [suppressed 5 similar messages]
W20240502 07:59:14.376977 31157 consensus_peers.cc:485] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 -> Peer 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701): Couldn't send request to peer 70e6a2ff133a488799712f7831863d51. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.65:39701: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 07:59:16.507953 31616 consensus_queue.cc:786] T e127013489864087b78e4e2f68290954 P d64238ec658a4dddaee6986e184db7d1 [LEADER]: Peer 70e6a2ff133a488799712f7831863d51 is lagging by at least 3692 ops behind the committed index  [suppressed 5 similar messages]
I20240502 07:59:16.660398 30029 linked_list-test-util.h:417] Sample at HT timestamp: 7023152155265122304 Inserted count: 208950
I20240502 07:59:16.660497 30029 linked_list-test-util.h:421] Finished inserting list. Added 208950 in chain
I20240502 07:59:16.660516 30029 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 07:59:16.660530 30029 linked_list-test-util.h:424] 0: -4080417125080825856
I20240502 07:59:16.660544 30029 linked_list-test-util.h:424] 1: 7219077484723306497
I20240502 07:59:16.660559 30029 linked_list-test-util.h:424] 2: 71546545841176578
I20240502 07:59:16.660575 30029 linked_list-test-util.h:424] 3: -7075702918064242685
I20240502 07:59:16.660595 30029 linked_list-test-util.h:424] 4: 143374566659063812
I20240502 07:59:16.660614 30029 linked_list-test-util.h:424] 5: -7004156372223066107
I20240502 07:59:16.660638 30029 linked_list-test-util.h:424] 6: 4295338237581066246
I20240502 07:59:16.660655 30029 linked_list-test-util.h:424] 7: -2852051963812773881
I20240502 07:59:16.660670 30029 linked_list-test-util.h:424] 8: 4366884783422242824
I20240502 07:59:16.660686 30029 linked_list-test-util.h:424] 9: -2780364680483176439
I20240502 07:59:16.660701 30029 linked_list-test-util.h:424] 10: 8518848454344245258
I20240502 07:59:16.660715 30029 linked_list-test-util.h:424] 11: 1371739727927115787
I20240502 07:59:16.660730 30029 linked_list-test-util.h:424] 12: 8590676475162132492
I20240502 07:59:16.660745 30029 linked_list-test-util.h:424] 13: 1443145536280002573
I20240502 07:59:16.660760 30029 linked_list-test-util.h:424] 14: -5703963190137126898
I20240502 07:59:16.660775 30029 linked_list-test-util.h:424] 15: 5595249944690294799
I20240502 07:59:16.660790 30029 linked_list-test-util.h:424] 16: -5632275906807529456
I20240502 07:59:16.660806 30029 linked_list-test-util.h:424] 17: 5666937228019892241
I20240502 07:59:16.660817 30029 linked_list-test-util.h:424] 18: -1480171498397237230
I20240502 07:59:16.660827 30029 linked_list-test-util.h:424] 19: -8627702437279367149
I20240502 07:59:16.660842 30029 linked_list-test-util.h:424] 20: -1408765690044350444
I20240502 07:59:16.660852 30029 linked_list-test-util.h:424] 21: -8555874416461479915
I20240502 07:59:16.660871 30029 linked_list-test-util.h:424] 22: 2743338718365941782
I20240502 07:59:16.660882 30029 linked_list-test-util.h:424] 23: -4403910745539477481
I20240502 07:59:16.660897 30029 linked_list-test-util.h:424] 24: 2815166739183829016
I20240502 07:59:16.660908 30029 linked_list-test-util.h:424] 25: -4332364199698300903
I20240502 07:59:16.660924 30029 linked_list-test-util.h:424] 26: 6967130410105831450
I20240502 07:59:16.660934 30029 linked_list-test-util.h:424] 27: -4260676916368703461
I20240502 07:59:16.660944 30029 linked_list-test-util.h:424] 28: 7038676955947008028
I20240502 07:59:16.660954 30029 linked_list-test-util.h:424] 29: -108572507958411235
I20240502 07:59:16.660964 30029 linked_list-test-util.h:424] 30: -7256103446840541154
I20240502 07:59:16.660975 30029 linked_list-test-util.h:424] 31: -36744487140524001
I20240502 07:59:16.660985 30029 linked_list-test-util.h:424] 32: -7184275426022653920
I20240502 07:59:16.660995 30029 linked_list-test-util.h:424] 33: 4114937708804767777
I20240502 07:59:16.661005 30029 linked_list-test-util.h:424] 34: -3032311755100651486
I20240502 07:59:16.661015 30029 linked_list-test-util.h:424] 35: 4186765729622655011
I20240502 07:59:16.661026 30029 linked_list-test-util.h:424] 36: -2960483734282764252
I20240502 07:59:16.661036 30029 linked_list-test-util.h:424] 37: 8338729400544657445
I20240502 07:59:16.661046 30029 linked_list-test-util.h:424] 38: 1191339199150817318
I20240502 07:59:16.661055 30029 linked_list-test-util.h:424] 39: 8410557421362544679
I20240502 07:59:16.661065 30029 linked_list-test-util.h:424] 40: 1263026482480414760
I20240502 07:59:16.661075 30029 linked_list-test-util.h:424] 41: -5884222981425004503
I20240502 07:59:16.661093 30029 linked_list-test-util.h:424] 42: 5415130890890706986
I20240502 07:59:16.661104 30029 linked_list-test-util.h:424] 43: -5812676435583827925
I20240502 07:59:16.661123 30029 linked_list-test-util.h:424] 44: 5486818174220304428
I20240502 07:59:16.661139 30029 linked_list-test-util.h:424] 45: -1660572027173535699
I20240502 07:59:16.661154 30029 linked_list-test-util.h:424] 46: -8807821491078954962
I20240502 07:59:16.661170 30029 linked_list-test-util.h:424] 47: -1588884743843938257
I20240502 07:59:16.661183 30029 linked_list-test-util.h:424] 48: -8736415682726068176
I20240502 07:59:16.661195 30029 linked_list-test-util.h:424] 49: 2563219664566353969
I20240502 07:59:16.697528 30029 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
/tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.65:39701
--local_ip_for_outbound_sockets=127.29.83.65
--tserver_master_addrs=127.29.83.126:33021
--webserver_port=46637
--webserver_interface=127.29.83.65
--builtin_ntp_servers=127.29.83.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 07:59:16.812526 31630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:16.812768 31630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:16.812844 31630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:16.816087 31630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:16.816218 31630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.65
I20240502 07:59:16.820048 31630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.65:39701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.29.83.65
--webserver_port=46637
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.29.83.126:33021
--never_fsync=true
--heap_profile_path=/tmp/kudu.31630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.65
--log_dir=/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:16.821002 31630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:16.822154 31630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:16.831338 31635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.832000 31636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:16.832561 31638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:16.833307 31630 server_base.cc:1034] running on GCE node
I20240502 07:59:16.833703 31630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:16.834316 31630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:16.835501 31630 hybrid_clock.cc:648] HybridClock initialized: now 1714636756835480 us; error 40 us; skew 500 ppm
I20240502 07:59:16.837493 31630 webserver.cc:460] Webserver started at http://127.29.83.65:46637/ using document root <none> and password file <none>
I20240502 07:59:16.838057 31630 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:16.838127 31630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:16.841652 31630 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:59:16.843899 31644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 07:59:16.845039 31630 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 07:59:16.845172 31630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
uuid: "70e6a2ff133a488799712f7831863d51"
format_stamp: "Formatted at 2024-05-02 07:58:45 on dist-test-slave-2kd1"
I20240502 07:59:16.845638 31630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:16.861300 31630 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 07:59:16.862015 31630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:16.862182 31630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:16.862809 31630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:16.864256 31651 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 07:59:16.867641 31630 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 07:59:16.867722 31630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20240502 07:59:16.867785 31630 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 07:59:16.869412 31630 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 07:59:16.869460 31630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20240502 07:59:16.869747 31651 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Bootstrap starting.
I20240502 07:59:16.880082 31630 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.65:39701
I20240502 07:59:16.880126 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.65:39701 every 8 connection(s)
I20240502 07:59:16.881131 31630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/data/info.pb
I20240502 07:59:16.883144 31651 log.cc:824] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:16.884238 30029 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu as pid 31630
I20240502 07:59:16.884774 31651 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.885213 31651 log.cc:1225] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskQ7SBdA/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714636688565791-30029-0/linked-list-cluster/ts-0/wal/wals/e127013489864087b78e4e2f68290954.recovery-1714636756885159
I20240502 07:59:16.885265 31651 tablet_bootstrap.cc:492] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Bootstrap complete.
I20240502 07:59:16.885509 31651 ts_tablet_manager.cc:1367] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.006s
I20240502 07:59:16.888382 31651 raft_consensus.cc:348] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:59:16.888815 31651 raft_consensus.cc:729] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e6a2ff133a488799712f7831863d51, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.889355 31651 consensus_queue.cc:260] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:59:16.890290 31651 ts_tablet_manager.cc:1398] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 07:59:16.890636 31759 heartbeater.cc:347] Connected to a master server at 127.29.83.126:33021
I20240502 07:59:16.890869 31759 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:16.891451 31759 heartbeater.cc:511] Master 127.29.83.126:33021 requested a full tablet report, sending...
I20240502 07:59:16.893419 30950 ts_manager.cc:194] Re-registered known tserver with Master: 70e6a2ff133a488799712f7831863d51 (127.29.83.65:39701)
I20240502 07:59:16.894361 30950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.65:47309
I20240502 07:59:16.896534 31759 heartbeater.cc:503] Master 127.29.83.126:33021 was elected leader, sending a full tablet report...
I20240502 07:59:16.904506 30029 cluster_itest_util.cc:256] Not converged past 4179 yet: 1.4210 1.4210 0.0
I20240502 07:59:17.007622 30029 cluster_itest_util.cc:256] Not converged past 4179 yet: 1.4210 1.4210 0.0
W20240502 07:59:17.132076 31760 tablet.cc:2369] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:17.210214 30029 cluster_itest_util.cc:256] Not converged past 4179 yet: 1.4210 1.4210 1.495
I20240502 07:59:17.512548 30029 cluster_itest_util.cc:256] Not converged past 4179 yet: 1.4210 1.4210 1.1450
I20240502 07:59:17.630460 31770 mvcc.cc:205] Tried to move back new op lower bound from 7023152046545408000 to 7023152032006541312. Current Snapshot: MvccSnapshot[applied={T|T < 7023152045231050752 or (T in {7023152045235478528,7023152045295108096,7023152045328691200,7023152045393719296,7023152045424136192,7023152045360840704,7023152045469171712,7023152045499514880,7023152045532131328,7023152045605847040,7023152045636820992,7023152045665525760,7023152045698011136,7023152045726752768,7023152045751775232,7023152045776658432,7023152045802061824,7023152045831479296,7023152045859856384,7023152045886713856,7023152045913133056,7023152045938372608,7023152045962567680,7023152045989568512,7023152046015942656,7023152046040465408,7023152046068543488,7023152046094041088,7023152046122770432,7023152046148956160,7023152046207922176,7023152046236844032,7023152046265753600,7023152046295138304,7023152046324002816,7023152046352662528})}]
I20240502 07:59:17.915205 30029 cluster_itest_util.cc:256] Not converged past 4179 yet: 1.4210 1.4210 1.2899
I20240502 07:59:18.417860 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 31141
I20240502 07:59:18.427165 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 31272
I20240502 07:59:18.437170 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636736650811 usec, L: 1 Expected Rows: 69500
I20240502 07:59:18.438122 30029 meta_cache.cc:1510] marking tablet server d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) as failed
W20240502 07:59:18.438201 30029 meta_cache.cc:302] tablet e127013489864087b78e4e2f68290954: replica d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
I20240502 07:59:18.535359 30029 linked_list-test-util.h:528] Calling callback on tserver 70e6a2ff133a488799712f7831863d51
I20240502 07:59:19.680040 31782 raft_consensus.cc:482] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d64238ec658a4dddaee6986e184db7d1)
I20240502 07:59:19.692054 31782 raft_consensus.cc:504] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:59:19.693436 31782 leader_election.cc:288] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375), c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779)
W20240502 07:59:19.701176 31646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
W20240502 07:59:19.705003 31646 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
W20240502 07:59:19.705824 31648 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779): Network error: Client connection negotiation failed: client connection to 127.29.83.67:43779: connect: Connection refused (error 111)
I20240502 07:59:19.705935 31648 leader_election.cc:302] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70e6a2ff133a488799712f7831863d51; no voters: c466253287d7465e80c44f03eba6ceb8, d64238ec658a4dddaee6986e184db7d1
I20240502 07:59:19.706350 31782 raft_consensus.cc:2726] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:59:19.856668 30029 linked_list-test-util.h:767] Done collecting results (69500 rows in 1321.4ms)
I20240502 07:59:19.928516 30029 linked_list-test-util.h:556] Snapshot: P: 1714636736650811 usec, L: 1 verified. Result: OK
I20240502 07:59:19.930565 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636746651459 usec, L: 1 Expected Rows: 139050
I20240502 07:59:19.931465 30029 meta_cache.cc:1510] marking tablet server d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) as failed
W20240502 07:59:19.931535 30029 meta_cache.cc:302] tablet e127013489864087b78e4e2f68290954: replica d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
I20240502 07:59:20.039670 30029 linked_list-test-util.h:528] Calling callback on tserver 70e6a2ff133a488799712f7831863d51
I20240502 07:59:21.926770 31787 raft_consensus.cc:482] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d64238ec658a4dddaee6986e184db7d1)
I20240502 07:59:21.926931 31787 raft_consensus.cc:504] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:59:21.927376 31787 leader_election.cc:288] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375), c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779)
W20240502 07:59:21.928551 31646 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
W20240502 07:59:21.928663 31648 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779): Network error: Client connection negotiation failed: client connection to 127.29.83.67:43779: connect: Connection refused (error 111)
I20240502 07:59:21.928732 31648 leader_election.cc:302] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70e6a2ff133a488799712f7831863d51; no voters: c466253287d7465e80c44f03eba6ceb8, d64238ec658a4dddaee6986e184db7d1
I20240502 07:59:21.928879 31787 raft_consensus.cc:2726] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:59:22.210455 30029 linked_list-test-util.h:767] Done collecting results (139050 rows in 2170.89ms)
I20240502 07:59:22.315897 30029 linked_list-test-util.h:556] Snapshot: P: 1714636746651459 usec, L: 1 verified. Result: OK
I20240502 07:59:22.317906 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 1714636756656524 usec, L: 1 Expected Rows: 208950
I20240502 07:59:22.318814 30029 meta_cache.cc:1510] marking tablet server d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) as failed
W20240502 07:59:22.318876 30029 meta_cache.cc:302] tablet e127013489864087b78e4e2f68290954: replica d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
I20240502 07:59:22.324687 30029 linked_list-test-util.h:528] Calling callback on tserver 70e6a2ff133a488799712f7831863d51
I20240502 07:59:23.944829 31792 raft_consensus.cc:482] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d64238ec658a4dddaee6986e184db7d1)
I20240502 07:59:23.944960 31792 raft_consensus.cc:504] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64238ec658a4dddaee6986e184db7d1" member_type: VOTER last_known_addr { host: "127.29.83.66" port: 39375 } } peers { permanent_uuid: "c466253287d7465e80c44f03eba6ceb8" member_type: VOTER last_known_addr { host: "127.29.83.67" port: 43779 } } peers { permanent_uuid: "70e6a2ff133a488799712f7831863d51" member_type: VOTER last_known_addr { host: "127.29.83.65" port: 39701 } }
I20240502 07:59:23.945385 31792 leader_election.cc:288] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375), c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779)
W20240502 07:59:23.946592 31648 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c466253287d7465e80c44f03eba6ceb8 (127.29.83.67:43779): Network error: Client connection negotiation failed: client connection to 127.29.83.67:43779: connect: Connection refused (error 111)
W20240502 07:59:23.946696 31646 leader_election.cc:334] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375): Network error: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
I20240502 07:59:23.946775 31646 leader_election.cc:302] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70e6a2ff133a488799712f7831863d51; no voters: c466253287d7465e80c44f03eba6ceb8, d64238ec658a4dddaee6986e184db7d1
I20240502 07:59:23.946902 31792 raft_consensus.cc:2726] T e127013489864087b78e4e2f68290954 P 70e6a2ff133a488799712f7831863d51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:59:24.334197 30029 linked_list-test-util.h:767] Done collecting results (208950 rows in 2009.59ms)
I20240502 07:59:24.496361 30029 linked_list-test-util.h:556] Snapshot: P: 1714636756656524 usec, L: 1 verified. Result: OK
I20240502 07:59:24.498399 30029 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 208950
I20240502 07:59:24.499357 30029 meta_cache.cc:1510] marking tablet server d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) as failed
W20240502 07:59:24.499423 30029 meta_cache.cc:302] tablet e127013489864087b78e4e2f68290954: replica d64238ec658a4dddaee6986e184db7d1 (127.29.83.66:39375) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.29.83.66:39375: connect: Connection refused (error 111)
I20240502 07:59:24.502274 30029 linked_list-test-util.h:528] Calling callback on tserver 70e6a2ff133a488799712f7831863d51
I20240502 07:59:26.521198 30029 linked_list-test-util.h:767] Done collecting results (208950 rows in 2019ms)
I20240502 07:59:26.682727 30029 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240502 07:59:26.683079 30029 linked_list-test-util.h:692] Successfully verified 208950 rows
I20240502 07:59:26.683585 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 31630
W20240502 07:59:26.683598 31404 client.cc:2025] Couldn't close scanner 26681e3a2cca494e85bd8583f52e23f2: Service unavailable: reactor is shutting down (error 108)
I20240502 07:59:26.693742 30029 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ7SBdA/build/debug/bin/kudu with pid 30919
2024-05-02T07:59:26Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (41273 ms)
[----------] 2 tests from LinkedListTest (78144 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (78144 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST